Theodo

How I Made My Life Easier by Letting QA Write E2E Tests?

Mon, 20 Sep 2021 00:00:00 GMT

Lately, I have worked as a web developer with a company that deals with logistics. Their business is to supervise the transport of commodities from point A to point B. The application I worked on was intended for the suppliers of this company to track their merchandise. Each persona would connect to the application and perform tasks such as declaring the dispatch or the reception of the products.

The people who worked on the previous projects followed a V-model lifecycle. These projects were delivered with many bugs after a few years. Developers spent a couple more years trying to fix the application. Therefore, the quality of the application is one of their main focuses. We were using an agile methodology and after each sprint, all the application was tested to ensure that no bug or regression appeared. QA (Quality assurance) was responsible for the quality of the application.

Testing the application - how to do it effectively?

The QA was manually testing the application. Running the full set of tests would take half a day, so the tests were performed every other week. It was not frequent enough to catch all the bugs and meanwhile, QA could not focus on their other tasks. It was not satisfying enough for the client and the QA,that is why the developer team suggested implementing end-to-end (e2e) testing.

But this idea was not perfect: in typical e2e testing, developers are writing the e2e tests. While they are implementing those tests, they are not developing features, which is detrimental for the client. On top of that, the QA, which is responsible for the quality of the application, does not understand what the e2e tests are testing.

Our challenge was finding a solution where QA would not have to test the application manually but without making testing the developers’ responsibility.

What if QA could write tests?

If the QA can write the e2e tests, the tests and the quality of the application are directly managed by the QA. The QA ensures thus the high quality of the application. It is pretty fast for the QA, because they write the test once, and it is then run automatically. And no need to wait two weeks now to run the tests anymore, they can be run every day.

Which means developers can fix bugs faster, for they might have created the bug recently and the code is still in their mind.

This was nice and good, but we had one major issue, for our QA did not have a developer background, yet popular e2e test frameworks, such as Cypress, are asking for knowledge in Javascript.

Writing tests without coding (or very little)

A common test framework, called Cucumber, can be used without development skills and both developers and the QA had already used it. Through its language (Gherkin), it allows team members to write tests in a human-readable language.

Cucumber can be then connected to many common testing tools, and it can be connected to Cypress thanks to cypress-cucumber-preprocessor. To develop tests, all you need is a grammar of test sentences, for example, “When I log in”. You still have to code, but your sentences are reusable, so when the grammar is done right, developers are not needed for every test case.

Our process was as follows. First, the QA would decide which test cases were to be tested automatically (critical test cases that are very tedious to run manually). The developers would then translate from Gherkin to Cypress by creating the Gherkin grammar that the QA can use.

You'll find an example of a feature written in Gherkin by the QA and the steps definitions in Cypress (as the developer will code them). You can retrieve this example here.

# Scenario in Cucumber/Gherkin
Feature: Cypress click
    Scenario: Click and navigate
        Given I'm on the "/" page
        When I click on "Querying"
        Then I should have "querying" in the URL

// Translation of the Gherkin steps to Cypres
import { Given, When, Then } from "cypress-cucumber-preprocessor/steps";

const baseUrl = "https://example.cypress.io";
Given("I'm on the {string} page", (url) => {
  cy.visit(baseUrl + url);
});

When("I click on {string}", (label) => {
  cy.get(".home-list").contains(label).click();
});

Then("I should have {string} in the URL", (url) => {
  cy.url().should("include", url);
});

Make our cucumber tests run

Our team was using Xray, a popular plugin that powers up Jira by providing tools to manage tests. As a result, the QA had to write all its test cases in XRay, which allows the QA to write manual tests or cucumber tests. In addition, Xray comes with an API that developers can use to get test cases or send tests results. Each morning, the tests were automatically pulled from XRay, run and then the results were sent to XRay.

Example of a test case in XRay

How was this solution for us?

With this solution, developers do as little code as possible. They only need to translate the Gherkin sentences into Cypress tests, but each sentence is reusable, so the QA can write new tests without the need of development.

The QA has a complete picture of the quality of the application. They decide which test cases should be executed each day and they know immediately if a test fails.

This process is not perfect and it still requires some maintenance from the developers, but no more than for regular automated tests. It even requires less time from developers as they are not responsible for keeping the test cases up-to-date.

And if you want to have your QA writing the e2e tests?

First, train your QA for using Cucumber. It is easy to understand, even with no development skills. Feel free to check the comprehensive Cucumber documentation.

Next, you need to decide where the QA should write their tests. Requiring QA to use git may be excessive. Xray is a good tool as it is designed to be used with Cucumber, but it can be any tool that your QA uses to write its test sets. Ideally, the tool should support Gherkin syntax and have an API to let developers run tests automatically.

Architecture guidelines for large Angular applications

Wed, 15 Sep 2021 00:00:00 GMT

TLDR;

When coding an Angular app for the first time, this is dangerous to keep old habits and reproduce patterns that worked for other apps like React or Vue. Working in a very large (~100 devs) project, I know that old habits can make code of poor quality.

I propose some guidelines inspired from Angular's recommendations and Clean Code practices that will help your team to write maintanable code and ease newcomer's onboarding!

Put the code right where it belongs : use the power of services and pipes to unclog your components.
Code in a reactive way your components with services
Avoid passing inputs/outputs all around thanks to the Dependency Injection System.
Split Services when they become too big. Do the same with modules

Old habits that get you wrong

Those patterns you need to let go of

Angular's intimate relationship with Typescript makes it easy to put data logic everywhere. Therefore it's easy to agree that dumb contains view-printing, and smart contains fetching. It is a sound decision to improve your code readability.

In terms of performance and seperation of duties, Angular proposes an idiomatic way to code. But many of us have been tempted to try famous patterns like Container Components (or smart), or trying to reproduce stateless Components of React and other frameworks.

That being said, Data often plays a central role. Once again, the temptation to try famous patterns like functional components and a Redux data-store is very strong. Indeed, some don't see how to find a robust way to manage their data.

Nevertheless, trying to mimick patterns you learned on other codebases blindly into an Angular application is such a dangerous idea and I want to show you how to make your Angular application maintanable.

Improving readability in consulting companies

I have been working here at Theodo (also in the UK) with teams that encoutered a huge developers turnover. Developers rarely stayed more than one year on the same code base. I even found myself switching teams every 2 months.

Therefore, it is crucial to choose a pattern that will help new developers to quickly switch codebases. That's why I want to introduce you to the right way of thinking your Angular Application Architecture. This approach makes small use of other famous patterns and mainly explains Angular's recommendations.

Guidelines for clean and efficient Angular coding

Use angular-cli

These are the rules you can establish in your team to make you Angular application cleaner. And as a quick rule, I advise you to always use angular-cli when creating new components and services in your app, so you're not tempted to write less.

You are writing presentation code in .ts or .html ? Make a pipe

If the data needs simple mappers, create pipes that you will use in component's template. Indeed those mappers add usually nothing but noise to your code, hide them away !

Don't write functions for data transformation in your component's .ts, and don't write complex functions in your component's html template. Code smells :

You call functions in your template (See why that's a bad idea)
You write a lot of logic in {{ your template }}

Here is an example of a too-intelligent component. It has functions in its template, with logic...

<ul>
  <li *ngFor="let member of team.member">
    {{ isUserFrench(member) ? '🇫🇷' : '🚩' }} Star: {{
    isUserCapableOneOfTheBest(member) }}
  </li>
</ul>

You can put everything in a pipe, a test it seperately !

@Pipe({
  name: 'memberBasicInfoList'
})
export class MemberBasicInfoListPipe implements PipeTransform {
  transform(team: Team): {flag: '🚩' | '🇫🇷', isStar: boolean}[] {
     //calculate flag and isStar
     return {flag, isStar}
  }

Then you have a very readable template !

<ul>
  <li *ngFor="let memberInfo of team | memberBasicInfoList">
    {{ memberInfo.flag }} Star: {{ member.isStar }}
  </li>
</ul>

Make reactive Services that support your components business logic

Services are your intelligent blocks of code. If you wanted to put code in a container component, put it here ! Here are some some rules of thumb :

Wrap all the data fetching in a service, and inject them in the component(s). This service should expose ready to use observables. It should also give methods to send data from components to the service. Same rule applies if a service becomes too big : split it and inject one service into the other service.

Code in a reactive way with observables as inputs and funtions as outputs

Use observables in your components as if they were already containing the data you want, because that is not the role of component to update the data. Then build public methods in your service to notify it of the last events. Your component should contain very minimum logic.

Think of your component as a puppet of the service. It shows the data the service wants. It tells the service when something happens.

Split your services when they become bigger than 100 lines

Put business logic as soon as possible in services and prepare the data inside of them to be displayed. For example a TeamMember.service that relies of TeamsStore.service and for example another User.service (that can come from somewhere else than TeamsModule)

To split your code, detect parts of code that are heavily linked to each others.

When your base module becomes too big and handles many topics : split into smaller modules

Put common business items in a module, for example "TeamsModule". Split them conviniently with Angular routing as soon as possible. When ?

Some parts become more and more independant, and some parts are never displayed together
Your module is too big for a new-comer to understand it

Your parent/children modules can communicate through common services. Learn more about when to split modules here : Angular: Understanding Modules and Services

Let's study an example: Breaking down one component into smaller simpler pieces

We talked about writing reactive services, and breaking down big chuncks of code into smaller meaningful one. This example shows a typical example when you have a very big component that needs to be refactored. I often found myself in this exact situation.

InviteTeamMemberComponent(httpClient: HttpClient, errorService: ErrorService, router: Router)
A. Can call the team-backend and POST a new team member
B. Will show an error message (in global snackbar) if failed
C. Will redirect to Team dashboard if successful
D. Creates FormGroup for team member informations (also stores the data and validates it)
E. Displays the data in the template
F. Keeps track of the loading state of the query

That's a lot of responsibilities, huh ?

Let's create a client that can do http-calls, especially the POST (task A.)

TeamMemberClient(httpClient: HttpClient)
A. Can call the team-backend and POST a new team member

and inject it in another service which has the responsibility to manipulate data (tasks B, C and D).

InviteTeamMemberService(teamMemberClient: TeamMemberClient, errorService: ErrorService, router : Router)
B. Will show an error message (in global snackbar) if failed
C. Will redirect to Team dashboard if successful
D. Creates FormGroup for team member informations (also stores the data and validates it)

that we inject inside the initial lighter component that does the two last UI features (tasks E and F)

InviteTeamMemberComponent(inviteTeamMemberService: InviteTeamMemberService)
E. Displays the data in the template
F. Keeps track of the loading state of the query

Avoid Input/Output and prefer injecting specialized services

You can sometimes avoid Input/Output drilling (more specialized article) with either service or content injection, this will look like a simple advice to Redux advocates, but here is the good news : we don't need redux !

Here is an example of input drilling where a 2 level deep button opens a modale upper in the hierarchy, all managed by a smart ancestor component !

Valuable code is spread among all the components and useless Input/Output mechanism pollutes the intermediary. On top of that, the ancestor component contains the data, and has the responsibility to update it. Here's what you would prefer :

This was an example of refactoring using service. You can also avoid one layer of Input/Output with content projection. In the example above, you could get rid of @Output() edit if you directly inject the button of the action.

What are the advantages of doing so ?

Synchronization within your team : You can still use old patterns, but don't make it a must ! Just communicate on your rules with your team and write coding guidelines that every one of you has read. You'll see tremendous improvement very quickly
Simpler hierarchy : your DOM is not polluted with useless container components. CSS is simpler, and inspecting the application is simpler too ! You can understand it faster ! If other teams use the same coding guidelines, you'll be able to switch teams easily.
Easier onboarding, as a new developer you know where things are by opening a package : services, components and pipes are doing what you expect them to do ! Also, less wrappers means less clicks to get to meaningful code !

AWS Fargate: harness the power of serverless for long-running computational tasks

Tue, 14 Sep 2021 00:00:00 GMT

TLDR: We all know the advantages of serverless computing: pay only for what you use; scale up and down with ease; abstract away the complexities of managing servers. In this article, I illustrate how you can acquire these benefits for long computational tasks which are too large for Lambda or Google Cloud Functions to handle. I share a case study from a recent project and how we used AWS Fargate (along with S3, Lambda, DynamoDB, and more!) to create a robust task architecture.

Recently, a client approached us with a state-of-the-art modelling system which had been designed by industry-leading domain experts. The techniques used in the modelling software meant that, for its use-case, the software could run simulations in an order of magnitude less time than the competing solutions.

Despite being an industry-leading scientific model, the delivery of that solution through software was holding the product back from its full potential. The software interface was a Python GUI and was distributed as source code directly to users. The model could take hours to run and if it failed (often due to invalid input), the modeller would only discover this error on returning to the program after several hours away, only to correct the error and rerun the test case. Often, multiple variations need to be simulated, however the desktop application couldn't execute multiple runs concurrently (as the software is limited by the processing power of the machine it runs on).

The mission

We wanted to bring this software to the market as an industry-leading product in terms of scientific modelling and software engineering. We would leverage cloud services to mitigate some of the disadvantages of the desktop implementation.

Here are some of the main things we wanted to add to the current functionality of the model:

Improved distribution by migrating to the cloud:
- Easier on-boarding (no software installation - just login).
- Improved security (no user access to source code).
- Run the model from any computer.
Increased scalability:
- Execute multiple model runs concurrently to save users' time.
Improved user experience:
- More data validation rules to prevent failed model runs.
- Email notifications for when the model has completed a run.

The challenge

The abstract problem to solve was that we had a 500,000-line black-box program that was tightly coupled to the environment and file system of the machine it was running on. In our case, three issues stand out:

The model code is tightly coupled to the modeller's development environment - it calls os.system in many places to run shell commands on specific libraries (these included dependencies on GNU Fortran - not something you see in your everyday web application!).
The model code is tightly coupled to the modeller's file system - it creates intermediate output files to save data between different steps in a run. This decision was initially made from a scientific standpoint - a modeller might want to check some intermediate output - and had stuck as a design pattern across the software.
The model code has significant complexity, and re-engineering it came with risks to the validity of the results.

These constraints meant that we needed:

To run the model code on a system with the same environment as the modellers' computers.
A way to persist intermediate files in the cloud between different runs.

The strategy

The high-level flow of a modeller executing a model run would go like this:

The modeller goes to the web application from their browser and configures a new run. Parameters that they configure are saved to DynamoDB and input files they upload are saved in S3.
The modeller clicks the "execute" button for a specific task, which triggers (a Lambda to trigger) a task in Fargate. This spins up a Docker image registered in ECR with the correct environment and source code to run the model. The task name is passed in as an environment variable.
The task pulls the correct input files from S3 and the user-defined parameters from DynamoDB.
The model task is then executed as it would be on a modeller's local computer.
The model syncs the resulting output directory to S3 and EventBridge and SES are used to notify the user of the status of the run via email.

The architect of the project drew a handy architecture diagram to illustrate exactly how we achieved these steps and I'll go through the steps that we took along the way.

Dockerize the tasks: Fargate + ECR + VPC

The first step was to create an environment to run the model; Docker was designed to solve this very problem ("well, it worked when it ran on my computer!").

A new container was defined and the required dependencies for the project were resolved (we used Poetry to manage the Python libraries).

We used Docker volumes for the project so that any intermediate files created within the container are synced with our outer file systems to inspect and validate. A docker-compose.yml file was also written to simplify bringing the container up and down and passing in environment variables from a .env file.

There are several discrete tasks that the model can run, each one generating different output files. For our use-case, we would have a single Docker image, which contains the model code for every type of task - the task to be executed would be defined as an environment variable (a container is designed to be brought up with a single purpose, then taken down once the task is complete).

Now that we know the task we want to execute, we can define the functions we want to run for each task. We define a TASK_DEFINITIONS variable which stores handlers for each task: preprocessor, executor (where we will run the main task), postprocessor, and failure (the model can be temperamental to user input, and we need to handle failure explicitly). I will come back to the pre- and post-processors in the next section.

The executor is simply a function which makes a call to the model source code to execute the desired task. This means we can define an env variable - TASK_NAME - and run the model locally by accessing bash from within the container:

docker-compose up -d
docker-compose exec <image-name> bash
python main.py

Note: my local machine doesn't have the correct dependencies to run the model code, but the container that I run on my local machine does.

The container can now be migrated to the cloud so that the model code can be accessed from anywhere in the world without the need to manually distribute the source code.

The image is registered with AWS Elastic Container Registry (ECR) where the container can be stored and accessed (Model registry in our architecture diagram). A Virtual Private Cloud (VPC) is configured to accept requests to trigger an AWS Fargate task.

Fargate is "serverless compute for containers"; this means that our container isn't running in the cloud when we don't need it - AWS handles the provisioning of resources to "spin up" a container from our image at the moment we want to run a task. This means that we are only paying for what we use whilst still having the potential to scale ad infinitum (we can trigger multiple tasks concurrently and AWS will just spin up more instances for us in the cloud). The one drawback of this architecture is that Fargate requires some time to build the container when we request for our task to be run (this was around a minute for our (large) container).

You can see Fargate (Model task) sitting at the centre of our architecture diagram within the VPC in ECS.

We also set up Cloudwatch to create logs for each run to help us and the modellers debug issues with the model.

Syncing input and output directories: S3

Until now, the model works in the cloud so long as the required input files are present in the file system of the Docker image; however, input files are specific to the context of a run, and if we want to get meaningful output then we need to pass in the correct input files at the point that we execute a task.

For each task, a new instance of the image is spun-up, so any intermediate files are lost when the container is pulled down and the context is lost (this is fine, as we want different executions to remain distinct from each other).

We set up an S3 bucket to store the inputs and outputs of tasks, which means that the files which we expect to persist between different tasks can be synced between permanent storage from task to task.

Each task was saving its output to a different directory, so within our TASK_DEFINITIONS, we define an input_dir, from which we would pull the contents from S3 and a number of output_dirs to which we would push the output of the model. These outputs might be downloaded by the user or used by a different task as input files. This syncing of files happens in the preprocess and postprocess of each task.

Data model: DynamoDB

As well as input files, the model also depends on a range of input parameters that the user can define. In the desktop app, the program gets these input parameters by reading and parsing a local file called INPUT_PARAMETERS.txt.

Each model run entity has a corresponding entity within a DynamoDB table (in practice there are multiple types of entity available for the user to manipulate). The database follows the single-table design as described in The DynamoDB Handbook by Alex DeBrie and uses dynamodb-toolbox to model entities. The user can interact with these entities via the web application to set properties of the run.

We created a corresponding manager in the dockerized model code for each type of entity in the database (e.g. RunManager) which can get the entity and also perform simple updates on text fields (set error messages etc.). The entity is retrieved from DynamoDB which then overrides the variables which were previously being retrieved from INPUT_PARAMETERS.txt, which otherwise describes a generic run.

The model is now no-longer dependent on a local file for input of discrete parameters; but how does the user configure those variables to get a final valid output to their desired parameters?

Web Application: Lambda + Next.js

It's now time to plug the main pieces together.

We developed a Next.js frontend to interface with the remote model. This comes with all the advantages of developing UI using technologies that were designed to handle UI (i.e. not Python!). This meant our input methods were more interactive and our outputs more visual, as we were able to leverage existing libraries.

We added validation rules to the inputs (with warning messages) to prevent the user running the model with parameters that don't make sense. We accept input files as upload fields of several forms, which are pre-processed to validate that the values in the files were within the expected range before uploading to S3.

We built our backend REST API using AWS Lambda. As with Fargate, Lambda offers flexible scaling, has a nice integration for the back-office tool we created in Retool, and we didn't mind the infamous cold starts for our use case.

Another advantage of using Lambda was that we can use AWS CloudFormation (infrastructure as code solution) to provision our infrastructure from a serverless.ts file which defines all the resources required for an environment. If we wanted to create a new environment, running sls deploy <env-name> would provision the resources defined in the file instead of needing to interact with the AWS console.

Notifying model outcomes: EventBridge + SES

Until now, a specific model task can be triggered to run in the cloud where it will sync the relevant input files, pull the correct input parameters, run the task, and sync the output files back to S3. Finally, we need to notify the user of the outcome of the model (success + results/ error + logs).

In the model, we defined a RunEventsManager which could send an event via AWS EventBridge when the postprocess or failure handler was completed in the model. We added a Lambda in our backend which was triggered by an EventBridge event (because we were using CloudFormation, we just needed to define the event that triggers the lambda in serverless.ts).

When an event occurs, the Lambda uses AWS SES client to send an email to the user, which notifies them immediately when they can check their results - there's no need to come back to check that the model hasn't failed every 15 minutes anymore!

Summary

Using the described architecture, we had delivered a significant improvement to the user experience in running scientific modelling tasks by bringing the software to the cloud and affording it all the advantages of a serverless implementation.

Rendering Blender scenes in the cloud with AWS Lambda

Sat, 28 Aug 2021 00:00:00 GMT

You can fairly easily leverage AWS Lambda to render scenes with Blender. It can make sense to use Lambda functions if you need to render a large number of assets in little time and if each asset is simple enough to be loaded and rendered in reasonable time (within the maximum lifetime of a Lambda) given the limited resources offered by Lambda functions (maximum 6 vCPUs and 10GB of RAM at the time of writing). If you have to render more complex assets, going with EC2 instances or AWS Thinkbox Deadline is a better solution.

When I came up with the idea of running Blender on Lambda, I naturally checked if someone was doing it somewhere and found out that @awsgeek had done something similar in 2019 so at least I knew it was possible. What we are going to present in this article slightly differs from his approach though.

The architecture

Here is the very simple architecture that allows you to render Blender scenes on AWS Lambda stored on an S3 Bucket. The final renders are stored into the same bucket.

The files you will need to run this "at home" (in the cloud)

To run this on your own AWS account, you will need four files:

Dockerfile describes the container image that each lambda function execution is going to run
serverless.yml describes the infrastructure above
app.py handles the event from the Lambda infrastructure and starts up blender
script.py manipulates Blender from the inside to lad a scene and render it.

File structure

Here is how these files should be arranged:

project directory
+-- Dockerfile
+-- serverless.yml
+-- app
|   +-- app.py
|   +-- script.py

Dockerfile

For the Dockerfile, I started from a great set of Docker images for Blender maintained by the R&D team of the New York Times. We're using the latest image that runs Blender rendering with CPU only (as Lambdas don't have GPU capabilities unfortunately).

FROM nytimes/blender:2.93-cpu-ubuntu18.04

ARG FUNCTION_DIR="/home/app/"
RUN mkdir -p ${FUNCTION_DIR}
COPY app/* ${FUNCTION_DIR}
RUN pip install boto3
RUN pip install awslambdaric --target ${FUNCTION_DIR}

WORKDIR ${FUNCTION_DIR}

ENTRYPOINT [ "/bin/2.93/python/bin/python3.9", "-m", "awslambdaric" ]

CMD [ "app.handler" ]

Serverless configuration file

The serverless.yml file is fairly simple:

service: blender-on-lambda
frameworkVersion: '2'
configValidationMode: error

custom:
  destinationBucket: blender-on-lambda-bucket

provider:
  name: aws
  region: us-east-1
  memorySize: 10240
  timeout: 900
  ecr:
    images:
      blender-container-image:
        path: ./

  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "s3:PutObject"
        - "s3:GetObject"
      Resource:
        Fn::Join:
          - ""
          - - "arn:aws:s3:::"
            - "${self:custom.destinationBucket}"
            - "/*"

functions:
  render:
    image:
      name: blender-container-image
    maximumRetryAttempts: 0

resources:
  Resources:
    S3BucketOutputs:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: ${self:custom.destinationBucket}

In this example, I am using the largest available Lambda functions with 10GB or RAM and 6 vCPUs so that the rendering goes as fast as possible.

The Python Lambda handler

The app.py file contains the logic that will be triggered by the Lambda infrastructure and it will receive the event passed as a trigger. In this case, the event must contain the width and height of the image to be rendered.

import json
import os


def handler(event, context):
    os.system(f"blender -b -P script.py -- {event.get('width', 0)} {event.get('height', 0)}")
    return {
        "statusCode": 200,
        "body": json.dumps({"message": 'ok'})
    }

The Python Blender script

And finally, here is the python script that will be executed from inside Blender to do the actual rendering.

from datetime import datetime
import os
import sys

import boto3
import bpy


argv = sys.argv
argv = argv[argv.index("--") + 1:]

s3 = boto3.resource("s3")
BUCKET_NAME = "jrb-material-renderer-bucket"
filename = f"{datetime.now().strftime('%Y_%m_%d-%I:%M:%S_%p')}.png"

s3.Bucket(BUCKET_NAME).download_file("scenes/scene.blend", "/tmp/scene.blend")

bpy.ops.wm.open_mainfile(filepath="/tmp/scene.blend", load_ui=False)
bpy.context.scene.render.filepath = f"/tmp/{filename}"
bpy.context.scene.render.resolution_x = int(argv[0])
bpy.context.scene.render.resolution_y = int(argv[1])
bpy.ops.render.render(write_still = True)

s3.Bucket(BUCKET_NAME).upload_file(f"/tmp/{filename}", f"renders/{filename}")

Cost

In the us-east-1 region of AWS, the largest available lambda will cost you $0.01 per minute. The request charge (the amount charged for every invocation) is probably negligible in comparison. Also, bear in mind that the generous free tier of Lambda might cover your costs! Same for S3.

I hope this can help anyone facing the same need that I did.

If you're interested in offloading your Blender render jobs to the cloud, follow me on Twitter. I'll be posting more content in the coming weeks such as:

cost comparisons with using other cloud providers
the cost impact of using GPU-enabled EC2 instances instead of Lambda
leveraging AWS EFS instead of S3 to load the scene into Blender

If there are other topics you'd be interested in, let me know on Twitter! My DMs are open.

Sending Financial Data to Salesforce Without the Hassle Thanks to Codat and Amazon AppFlow

Wed, 11 Aug 2021 00:00:00 GMT

We’ve recently built a secure data pipeline for a customer called Founders First Capital Partners. They needed to automatically load large amounts of financial data into Salesforce, their CRM, in order to make investment decisions. We used Amazon’s managed data integration service — AppFlow — which takes care of creating a secure connection and sending data to Salesforce. To retrieve the data securely from several financial tools, we leveraged Codat. Codat's single API powers robust integrations to over 29 different financial tools including QuickBooks, Stripe, Plaid and Oracle NetSuite. Codat takes care of all the heavy lifting involved in these integrations — authorization, data standardization,ongoing connectivity — leaving us free to focus on how to best use this data.

The architecture

We designed a simple serverless architecture that fit our needs.

Benefits of using these tools

Why would you do this instead of “simply” writing the connection to the APIs?

Less code to write. All you have to do is write code that puts data in an S3 bucket and AppFlow will take care of sending it to Salesforce (or any other supported SaaS tool you choose).
All the usual benefits of serverless and managed services: you only have to focus on the business logic that matters to your use case. AWS ensures your solution is always up and running (with an SLA of 99.9%). There are no software updates or patching to worry about. It scales with your needs.
Enhanced security: for each SaaS that AppFlow connects to, AWS establishes a secure AWS PrivateLink which is a great security guarantee without any additional effort.
The standardization of data provided by Codat allowed us to manipulate data from various sources into one consistent data structure, which is a huge time-saver.

Trade-offs of our solution

AppFlow can sometimes throw error messages that are not very helpful, which can be frustrating. But, just like with other AWS products, their team is open to feedback and continuously improves AppFlow.
The community adoption of AppFlow still seems to be nascent (based on the number of Stack Overflow discussions on the topic), which is a bit of a shame given how helpful this service is. As a result, it can be a struggle to find resources to help you get up and running.

The combination of Codat with Amazon AppFlow allowed us to load large amounts of critical financial data into Salesforce in a very short amount of time, with minimal development effort. I hope you can also save time by using these amazing tools!

Thanks to Nate Ritter, CTO of Founders First Capital Partners, for giving us the opportunity to work on this small piece of serverless sorcery. Founders First Capital Partners are based in San Diego, CA and support diverse founder-led small businesses and those in low to moderate income areas by providing them with funding and acceleration services.

Hasura: GraphQL Without the Baggage.

Wed, 04 Aug 2021 00:00:00 GMT

Spoiler Alert: Star Wars references below. Proceed at your own risk...

The introduction of GraphQL back in 2015 garnered a lot of hype in the development sphere - allowing clients to describe exactly what data they require (preventing over-fetching unneeded information) and unifying multiple resources into a singular request. This substantially improved scalability and reusability of backend endpoints, and makes the development experience all-in-all more pleasant.

So you might be thinking to yourself: "Great. What's the catch?"

Well... Implementing GraphQL into your backend will inadvertently increase complexity - you'll need to dedicate time on defining types, efficient resolvers, mutators, queries, etc. If you're working as a full-stack developer, you might find that using GraphQL is kind of like taking a dollar from one pocket and putting it in another. The added complexity can start to outweigh the benefits, especially on smaller scale projects. On top of that, caching is complex with GraphQL. With no in-built caching capabilities, this often becomes a pain point for developers.

That's not to diminish or underplay the clear benefits of using GraphQL.

However, I want to use this opportunity to touch on some of the less talked about vices of GraphQL, and present Hasura as a possible solution to some of these concerns. Hasura is a GraphQL service that automatically generates GraphQL endpoints from your database schemas, taking out a lot of the complexities and manual work needed. Simply by connecting Hasura to your SQL database, it will generate schemas and resolvers using the existing tables and views. It's quite magical in how quick it is to get up and running...

GraphQL Complexities

Boilerplate & Initial Setup

As you'll usually find with most other large scale dependencies or libraries, the addition of GraphQL will require some level of initial setup and boilerplate, you're adding a new layer to both your frontend and your backend. You're probably going to need to add a GraphQL client, define the types, schemas, error handling, caching on the frontend level. On the backend side, you'll need to define multiple resolvers to fetch different views of the data (get all the data/get a single datapoint with an id), as well as defining mutations for creating new datapoints, editing existing data, and deleting data.

Now you could argue that these would all still be required if you used normal REST endpoints - and you would be right. However, on top of these, you'll need to deal with typing on the backend, caching (you can't rely on native HTTP caching like REST APIs), higher order components, and additional validators to catch some edge-cases that arise from defined schemas.

A solution commonly taken on the backend side, is to define boilerplate abstractions to take the repetition out of having to define basic resolvers/queries for new data classes you add into your application, but again this takes time to setup (plus you're adding more complexity). While this might not be an issue for larger projects that can reap the benefits of using GraphQL, it becomes a massive swaying point away for small projects.

Resolver Hell

Resolvers are the functions that resolve the values for any field in a schema. They can return primitive values, or objects that get further broken down into more primitive types. You'll find that these typically return values from databases or other APIs.

Now there are a number of databases and ORMs that come with GraphQL mappers that will automatically handle resolvers for you when you are defining your data in objects. For example, in a project that I worked on, we use GORM, which handles resolving and schema generation. However, these do come with their overheads, and you would need to define your data models according to these data access tools.

In many cases you'll find that you're dealing with your database directly, and you'll be doing the DB querying yourself. This opens a whole new can of worms to deal with.

A common issue that many face is over-querying the database. Let's demonstrate this with an example:

You are creating an application to manage a large movie cinema company with multiple branches. You have a query that fetches a list of all the cinemas, the screens in each, the movies showing on each screen, and which production companies released them.

Now presume that there are 5 cinemas (branches), with 10 screens each, each playing 5 movies at any given point. This means that we have 1 query to get the cinemas, 5 queries for each cinema getting the screens, and 50 queries for the movies playing on each screen (50 calls to get the details of the movie playing on each screens). That comes down to a total of 56 database calls for this query.

Say you wanted to fetch some basic metadata about each of these movies as well (such as the production companies), that would be another 50 database calls added, which would increase our total to a whopping 106 database calls...

Cinema example above visualised.

This can be mitigated by strategising and coming up with efficient queries and resolvers, but the problem remains that it is very easy to fall into these common patterns that will result in overfetching. It's just the price that comes with allowing the client to specify their data needs and schemas.

There are solutions to deal with this - for example, Dataloader will batch some of these database calls together and cache some of the results. However, these solutions are far from perfect and they have limitations when making certain queries. Not to mention that by adding new dependencies, we start inadvertently increasing complexity of the application.

Performance & Caching

With complex views on the data, comes the cost of query complexity and the number of database calls increasing. It becomes a balancing game... You'll want to use GraphQL's ability to define graph-based traversals to fetch multiple resources (in fact, this is probably the single biggest selling point IMHO), but use it too much (or irresponsibly), and you might find that your queries become slow and start to massively affect performance.

Take the Star Wars example that is shown on the homepage of the GraphQL website:

type Query {
  hero: Character
}

type Character {
  name: String
  friends: [Character]
  homeWorld: Planet
}

type Planet {
  name: String
  climate: String
}

Let's try and visualise it with a flowchart, with the main trio of the original series. If Luke Skywalker is the hero of the original series (which is a false statement, because we all know the Ewoks were the true heroes), he has several friends, and each of these friends have a homeWorld.

GraphQL allows you to traverse the graph of Characters by defining resolvers to fetch the friends. Now this is a fairly simple example, and you'll find that real world data will usually have more complex relations and hence this graph will become larger. This will take a toll on performance, and you may find that certain queries will take unreasonably long to run.

A common approach taken to solve this problem is through caching, and although it won't work well for frequently changing data, this can help reduce the number of queries that need to be made regularly on data that rarely changes.

As an example, Apollo can allow you to define and control caching structures on your web applications, but if you have multiple frontend applications (say a web, Android, and iOS app), you'll probably need to handle caching on each of these levels separately. Again, added complexity that will be introduced by using GraphQL.

However, this options is only viable for data that isn't changing regularly... So in the case where Leia's home planet of Alderaan gets obliterated by the Death Star, or Han Solo and Leia become separated (ergo are no longer considered friends) due to their son falling to the dark side of the force, you may find that caching will result in data inaccuracies. This is where the importance of having well-written and efficient resolvers and database calls comes into play yet again.

And with that, all may seem lost, but from the darkness of caching, boilerplate, and complex resolvers comes the saving grace of Hasura...

Hasura

Hasura is an open source service that allows you to take your relational databases, and automatically generate GraphQL APIs on top of them. Effectively, this removes one of the biggest challenges that backend developers using GraphQL face, which is creating efficient SQL queries and resolvers.

Looking at the challenges posed above, the issue of excessive boilerplate code & high initial costs effectively become neutralised by using a tool like Hasura. I had a Postgres database running on my machine, and it took me a total of 15 minutes to spin up Hasura with Docker. Just by changing the docker-compose, I pointed it to my database, and it loaded up already connected and hosting a GraphQL endpoint with my defined views! Honestly, it felt like some form of vodoo magic! The steps I followed were really quite simple:

Fetch the docker-compose file with curl.
Run docker-compose up -d
Get yourself a cup of coffee while you wait for it to run.
Open the console page at localhost:8080/console
Click on the data tab, and click the "Connect Database" button.
Add a name for your database, the SQL flavour, and lastly the database URL.
Click the connect button.
After it's done connecting, click on the "GraphiQL" tab, and see that your database tables can be queried out of the box!

Now, a common question that may come up about the initial cost of using a technology is how well it integrates with existing codebases. Presume that you're already using a GraphQL endpoint for the backend of your application, and you want to incrementally introduce Hasura to handle the basic queries and subscriptions (yes, it comes prebuilt with subscriptions, and that can be an absolute pain with certain GraphQL servers). You can use a feature called Remote Schemas to point Hasura to your already existing GraphQL endpoint, and it'll automatically combine the schemas together to create a unified GraphQL endpoint that your frontend can use. Additionally, you can use this to delegate more complex business logic in your code to other services (such as a payment API).

The second challenge touched on above was the issue of inefficient and complex resolvers. Without getting into much detail, Hasura compiles down GraphQL queries using abstract syntax trees, validates and checks permissioning at this level, and then constructs the entire query into one raw SQL query. On top of this, they do 2 levels of caching (one at the SQL compilation level, and another at the database level with prepared statements) within their architecture to reduce the amount of times these compilation steps are required. They're building and improving on this architecture and performance with every update. For example, they've introduced JSON aggregations on the database level, effectively removing the need to flatten the results from the database calls in the backend - they claim this increases performance anywhere from 5-8 times! For those who are interested, they have an interesting article on their website talking about their architecture in more detail.

Hasura's Data Wrapper (Source: hasura.io)

Lastly, we touched on the issue of caching & performance. Within the data wrapper layer, there's already 2 levels of caching that will significantly improve performance. Additionally, there is another layer of caching, which is on the query level, that effectively allows you to cache the response being sent back to the client for a defined period of time. You can make a call with the @cached directive to cache the response, supplying an optional ttl argument (how long to cache the response):

query MyCachedQuery @cached(ttl: 120) {
  users {
    id
    name
  }
}

This caching comes working out of the box. In contrast to common caching approaches (such as using Apollo), having this caching done on the server side can be beneficial for unifying the responses on any given platform at any point of time. Imagine you have a cached query for the top players in a leaderboard. If the caches are set at different points on the client, there may be inconsistencies in how this data is displayed. Caching on the server-side circumvents this.

On top of that, Hasura's performance isn't only due to compiled queries, efficient resolvers and caching. There's a ton more work put into improving speed and performance and it is a regularly improved and maintained project that's already heavily utilised in the industry.

So to demonstrate the Star Wars example we saw above, I went ahead and created a database schema to have the following tables:

Character
Planet
friendsWith

I did all of this through the Hasura data admin panel (sort of like a lite version of PGAdmin). Through this data panel, you can choose which tables/views can be tracked by the GraphQL schema, and you can define table relationships and how they manifest in the GraphQL schema.

Switching to the "GraphiQL" tab, we can now see that the explorer now has a bunch of different datapoints:

You can fetch the list of each of the data types by calling the basic data type (for example, Planet), and this comes with pagination and filtering out of the box. Alternatively, if you're looking for some form of aggregation on the data, you can use the _aggregate datapoint, which can run some basic aggregation on the data (such as averaging a field, counting, finding the min/max the field, or even getting the standard deviation). Lastly, you can get a single data entry by using the by_pk variation.

Now, to see this in action, we construct a query asking for the first character in our character table. We ask for the name of the character, their home planet (along with its associated climate), and a list of their friends.

To be able to do this in the matter of a couple minutes is something that I still find magical.

I'll end here with some of the selling points that I think make Hasura really shine:

Automatically generates the schemas and resolvers for your GraphQL endpoints (less pain).
Comes prebuilt with option variables that you can use in your queries (for example, limit and offset, which will allow you to more or less handle lazy loading of datapoints).
Currently works with a number of SQL databases (Postgres, Microsoft SQL, Amazon Aurora, and Google Big Query) with more coming soon.
Easy to integrate with authentication and permissioning platforms, such as Auth0.
Gives you a web UI to manage the GraphQL endpoint, as well as an interface to manage the database and its views itself (think PGAdmin).
Using Remote Schemas, you can hook Hasura up with other GraphQL endpoints and other services to unify your endpoints.
Using Actions, you'll be able to define calls to custom handlers that will execute more complex business logic (which opens the door to serverless by using Lambdas)
You can define event triggers to call actions automatically (when a user signs up, send an email via a Lambda function).
Highly maintained and used by the community, with over 23k stars on Github and 250m+ downloads.

Conclusion

GraphQL provides some fascinating functionality that aims to make the data layer of applications unified and flexible. As with most things, this comes at a cost. The initial setup and boilerplates will mean that you will need to spend time upfront in creating the right infrastructure for using GraphQL. On top of that, the flexibility in schema means that you will need to have additional considerations when it comes to performance, and you will need to come up with efficient queries and resolvers. Lastly, with GraphQL we lose the native HTTP caching capabilities, meaning that caching is not standardized and becomes cumbersome additional work.

Hasura, acting as a service layer between your databases and clients, can automatically generate performant GraphQL queries, mutators, and even subscriptions, with minimal effort on the your end.

Whilst Hasura's solution to automatic generation of schemas and endpoints might not offer the full flexibility that large-scale highly complex applications might need, it opens the door to using GraphQL on smaller scale projects where the startup costs of introducing it would outweigh the benefits. I believe this is where Hasura will excel, and where it will ultimately find its place in the GraphQL market.

Writing a native Ionic plugin for Capacitor in less than 30 minutes

Mon, 12 Jul 2021 00:00:00 GMT

TLDR: When the pandemic first started, I decided to develop a contact-tracing mobile app. I was studying Computer Science at the time and decided to use the same cross-platform framework that we were using on our course - Ionic. I wanted to use Google's Nearby Messages API to share packets of information between iOS and Android devices via Bluetooth, but I couldn't find a plugin for this, so I decided to write my own!

View commits: Android plugin → iOS plugin → Import native plugins → Implementation

Ionic is a cross-platform mobile framework which allows you to develop an app using JavaScript/HTML/CSS and share this single implementation across different native devices. This Ionic project can then be compiled to native source code (Android/iOS) using Ionic's tool, Capacitor.

Using a cross-platform framework like this to develop your mobile app is almost always a good idea, because it means you can maintain a single codebase which runs on both Android and iOS. This means you're not duplicating business logic across two different codebases in two different languages with two different teams (to be avoided).

If you're looking to write some native code in a cross-platform project but you're still deciding which framework to use, check out this Theodo blog post on React Native as well!

Step 1 - Search for existing solutions

Firstly, let's have a quick check that one of the default plugins doesn't do the job for us already.

Secondly, let's search through the community plugins to find the feature we're looking for.

No luck finding an existing plugin? Don't worry - let's move on to step 2!

Step 2 - Start a new app with Ionic + Capacitor

I'm starting a new project from scratch just to show you how it's done.

npm install -g @ionic/cli

ionic start <app-name>
cd <app-name>
ionic serve <-- you can serve your project on localhost

The /src directory of our Ionic project holds the source code for our cross-platform app.

We can use Capacitor to compile this into two native projects:

npm install @capacitor/cli @capacitor/core

npx cap init
npx cap add ios
npx cap add android

When we make changes to our /src directory, we can apply those changes to our native projects by running npx ionic build && npx cap copy.

We now have our project set up and ready to add a custom plugin.

Step 3 - Check the documentation

Capacitor is well-documented and it's worth linking some references here to accompany this guide:

Getting started with Android

Android reference

Getting started with iOS

iOS reference

Step 4 - Define a strategy

There are three main steps to writing a plugin:

(iOS & Android) Write a class in each native project to hold the native code we want to run (this is our plugin).
(iOS & Android) Register these classes inside the bridge of our native projects to expose the methods.
(Javascript - Ionic) Import and call the methods of our registered plugin.

We should avoid duplicating code as much as much as possible (DRY), so all your business logic stays written in JavaScript, keeping the bare minimum written natively.

Your plugins should be narrow in scope (do one specific thing) - make sure to follow this philosophy and give your plugin a specific name ('Plugin' is a bad name!).

We have two main strategies for calling our plugins:

An (asynchronous) method: we call some native method which we can await in our business logic in Ionic.
Event listeners: define a listener in our business logic. Events can be triggered in our native code to pass payloads back to the business logic.

If we are subscribing to a stream of events (expecting our method to resolve with multiple payloads over a longer time period e.g. geolocation/ Bluetooth updates), then event listeners are the appropriate way to handle our native calls.

Step 5 - Write and register the Android plugin

[View the commit]

Open the /android directory of your project in Android Studio.

In /app/src/main/java/../../.. (next to MainActivity.java), we can create a new file (I'm calling mine IonicNativePluginExample.java) and write a class which extends the Capacitor Plugin class:

package /* <package-name> */; // e.g. io.ionic.starter

import com.getcapacitor.JSObject;
import com.getcapacitor.Plugin;
import com.getcapacitor.PluginCall;
import com.getcapacitor.PluginMethod;
import com.getcapacitor.annotation.CapacitorPlugin;

@CapacitorPlugin(name = "IonicNativePluginExample")
public class IonicNativePluginExample extends Plugin {

    @PluginMethod
    public void NativeMethod(PluginCall call){
        JSObject result = new JSObject();
        result.put("message", "Hello Android user!");
        call.resolve(result);
    }

    @PluginMethod
    public void NotifyListeners(PluginCall call){
        JSObject result = new JSObject();
        result.put("message", "Hello Android user!");
        notifyListeners("EVENT_LISTENER_NAME", result);
    }

}

Next, we can register this plugin in MainActivity.java where Capacitor initialises its bridge:

package /* <package-name> */; // e.g. io.ionic.starter

import android.os.Bundle;
import com.getcapacitor.BridgeActivity;

public class MainActivity extends BridgeActivity {
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        registerPlugin(IonicNativePluginExample.class);

    }
}

We can run our project like any other native Android project, by pressing the 'play' button in Android Studio.

If you're looking to develop in Kotlin - Android Studio comes with tooling to convert this Java class to Kotlin (right-click the file you want to convert).

Note: you might need to edit the Gradle distributionUrl to get the project build to succeed - there's an example in my repo here.

Step 6 - Write and register the iOS plugin

[View the commit]

Open the ios/App directory of your project in Xcode.

In /App (next to AppDelegate.swift), we can right-click on the directory to create a new swift file (I'm calling mine IonicNativePluginExample.swift). This will hold our plugin class with our IonicNativePluginExample method.

import Foundation
import Capacitor

@objc(IonicNativePluginExample)
public class IonicNativePluginExample: CAPPlugin {

	@objc func NativeMethod(_ call: CAPPluginCall) {
		call.resolve(["message": "Hello iOS user!"])
	}

	@objc func NotifyListeners(_ call: CAPPluginCall) {
		self.notifyListeners(
			"EVENT_LISTENER_NAME",
			data: ["message": "Hello iOS user!"]
		)
	}

}

Next, we must register our plugin in a new Objective-C file called IonicNativePluginExample.m (same name as our plugin file but with a .m extension). When prompted by Xcode, create a Bridging Header file (which is an empty file called App-Bridging-Header.h), then register the plugin like so:

#import <Foundation/Foundation.h>
#import <Capacitor/Capacitor.h>

CAP_PLUGIN(IonicNativePluginExample, "IonicNativePluginExample",
           CAP_PLUGIN_METHOD(NativeMethod, CAPPluginReturnPromise);
           CAP_PLUGIN_METHOD(NotifyListeners, CAPPluginReturnPromise);
)

We can run our project like any other native iOS project, by pressing the 'play' button in Xcode.

Step 7 - Import and call our plugin in Ionic

[View the commit]

Finally, we can now access the plugins we have written from our Ionic project. I've created a /plugins directory to export mine from. We import registerPlugin from Capacitor to find the plugin we have registered in Android and iOS:

import { registerPlugin, Plugin } from "@capacitor/core";

// we can take advantage of TypeScript here!
interface NativePluginInterface extends Plugin {
  NativeMethod: () => Promise<Record<"message", string>>;
  NotifyListeners: () => Promise<void>;
};

// it's important that both Android and iOS plugins have the same name
export const IonicNativePluginExample = registerPlugin<NativePluginInterface>(
  "IonicNativePluginExample"
);

Now we can call the methods in our plugin and access the native code:

import { IonicNativePluginExample } from './plugins/IonicNativePluginExample'

...

// add a listener to native events which invokes some callback
IonicNativePluginExample.addListener("EVENT_LISTENER_NAME", ({ message }) =>
    console.log(message);
);

// destructure the methods to call our native code from our non-native app
const { NativeMethod, NotifyListeners } = IonicNativePluginExample;

// native methods are asynchronous
const { message } = await NativeMethod();

// this method will trigger our event listener
NotifyListeners();

Note: to see our changes reflected when we run the project natively, we need to run npx ionic build && npx cap copy.

We now have a cross-platform app set up ready to write a native implementation. We run the same business logic from our Ionic app and get a different implementation for each device we run on.

I made a small edit to our Ionic project to create the simple implementation below - you can view the commit here.

More complex plugins

With the above setup, you should be ready to write more complex native code to suit your needs for a cross-platform Ionic application.

I wrote a native Plugin for Google's Nearby Messages API, which is a publish-subscribe API made by Google which facilitates the transfer of information between internet-connected Android and iOS devices.

Nearby Messages uses a combination of Bluetooth, Bluetooth Low Energy, Wi-Fi, and near-ultrasonic communication between nearby devices to create a unique pairing. This pairing is then used to send small payloads over the internet between nearby devices.

Here's a quick code snippet from the Android plugin:

...

import com.google.android.gms.nearby.Nearby;
import com.google.android.gms.nearby.messages.Message;
import com.google.android.gms.nearby.messages.MessageListener;

...

@NativePlugin()
public class NearbyMessagesPlugin extends Plugin {

		private Message mMessage;
    private MessageListener mMessageListener = new MessageListener() {
        @Override
        public void onFound(Message message) {
            JSObject result = new JSObject();
            result.put("message", new String (message.getContent()));
            notifyListeners("FOUND_MESSAGE", result);
        }

        @Override
        public void onLost(Message message) {
            JSObject result = new JSObject();
            result.put("message",  new String (message.getContent()));
            notifyListeners("LOST_MESSAGE", result);
        }
    };

    @PluginMethod
    public void Subscribe(PluginCall call){
        Nearby.getMessagesClient(getContext()).subscribe(mMessageListener);
    }

    @PluginMethod
    public void Publish(PluginCall call){
        String value = call.getString("message", "-");
        mMessage = new Message(value.getBytes());
        Nearby.getMessagesClient(getContext()).publish(mMessage);
    }
}

Event listeners are appropriate here because although we don't know when/if the phone will pick up a Bluetooth signal, a side-effect can still be triggered in the business logic when it does. Subscribe can be invoked to subscribe to mMessageListener and listen for Bluetooth messages (or we could just call Subscribe in the onCreate of our native code if we wanted to).

Summary

Once you've got a simple implementation, you can move on and write some more complex native code.

If you want a quick-start, you can fork my repository on GitHub, but following along with the article and doing it yourself will really help your understanding.

RisXSS, the missing ESLint rule for React and Vue

Mon, 05 Jul 2021 00:00:00 GMT

XSS attacks are expected to be ranked 3rd in the OWASP Top 10, compared to their 7th rank in 2021 (source). The pace of application creations is increasing and so is the surface of attacks, so making your applications secure is more and more challenging and needed. React and Vue are some of the most used frameworks to build applications and include built-in protection for some types of XSS by escaping user input by default. However, there may be some cases when they need to display user content as raw HTML (a blog post with style for instance).

export const BlogPost = ({ post }) => {
  return (
    <div dangerouslySetInnerHTML={post} />
  );
};

(All the examples in this article use React but are similar with Vue).

A first attempt to prevent XSS attacks

This opens a large hole in the XSS protection. A secure solution is to first sanitize the content with a library that will remove unsafe parts of the content while keeping useful tags and attributes (for styling purpose for instance). The most popular libraries for this are DOMPurify and js-xss.

This solution is not always known though. On a website I worked on, I saw the use of dangerouslySetInnerHTML that was not properly sanitized and introduced a real XSS vulnerability. My first reaction back then was: "How can I make sure this will never happen again?" So I searched the Internet and I happily found ESLint rules designed precisely to prevent the use of dangerouslySetInnerHTML and v-html. I added these rules on my projects and some time later, again, I found another XSS vulnerability.

How did it happen? Easy: some developer wanted to display sanitized content with style, so they used v-html while sanitizing the output first and added a comment to disable the ESLint rule for this line. This is correct in this case. Then, another developer copied-pasted (who does not?) this code in another page, but the content was not sanitized.

When you have no other means than to disable a rule to do something, disabling the rule becomes the norm, and the rule loses its purpose.

Custom ESLint rule to the rescue

This is when I talked with some colleagues and started a custom ESLint rule to warn about insecure HTML only if the HTML is not sanitized. Corentin Normand and Guillaume Klaus implemented the logic (and even wrote a tutorial about it) and it became RisXSS.

RisXSS warns you about insecure use of dangerouslySetInnerHTML and v-html but only if the content is not sanitized. The following snippet will not display a warning:

import { sanitize } from 'dompurify';

export const BlogPost = ({ post }) => {
  return (
    <div dangerouslySetInnerHTML={{ __html: sanitize(post) }} />
  );
};

By offering a smart ESLint rule, RisXSS achieves the best of both worlds:

preventing XSS
allowing developers to display content with style

Now that you understand the need of RisXSS, go check out how to install it on your project on the RisXSS homepage.

How to Choose Between Different Code Reusable Pattern in Vue.js?

Mon, 28 Jun 2021 00:00:00 GMT

As developers at Theodo, we are always looking for efficient ways to reuse our code without repeating ourselves (DRY principle). In the context of an e-commerce project using the Vue Storefront framework and Vue 2, we had the opportunity to study the best ways to reuse functional behaviors in Vue.

We found a lot of solutions in the Vue documentation such as Mixins, composition API and scoped slots. I hope this article will help you choose yours according to your projects.

Here are the key questions I will answer thereafter:

What is the best method to share logic among code in Vue 2?
What are the advantages and drawbacks of using the newly developed composition API, scoped slots, or Mixins to reuse some logic?
When should you use one over the other?

TL;DR:

Using Mixins is one of the ways to manage shareable logic in Vue. But they have many drawbacks and should be avoided in both Vue 2 and Vue 3.
Scoped slots are a great alternative to Mixins for Vue 2 and they still have their place in Vue 3.
The Vue 3 composition API is a powerful tool to create readable component shareable logic. It is the option I would recommend to use in most cases but is for now limited to Vue 3.

Context

We had the responsibility to develop different authentication connections for a web platform (Google & Facebook OAuth in addition to a basic email/password authentication).

In short, we needed to create three buttons that trigger three different actions. Our buttons looked very similar so we wanted to avoid duplicating the UI code. We thus created a single LoginButton.vue taking 2 props: label and icon. The button triggers an event when clicked (event named click).

// components/LoginButton.vue

<template>
  <button @click="$emit('click')">
    <img v-if="icon" :src="icon.src" alt=""/>
    <span>{{ label }}</span>
  </button>
</template>

<script>
export default {
  props: {
    icon: {
      type: Object,
    },
    label: {
      type: String,
      required: true
    }
  }
}
</script>

Our objective was to:

build the three authentication processes using this generic login button.
organize our three different login logics in a way that would allow them to be used anywhere in the app without repeating ourselves.
keep the code as readable as possible

Before even starting, we wondered if creating basic js functions for each connection methods in a separate JS file would be enough. The issue is that the implementation of an oauth connection requires access to the component lifecycle hooks (the initialization of the oauth client has to be triggered in the mounted hook). This access to the component options thus became a new requirement in our quest for the best solution.

First lead: Define the different connection methods within a wrapping component

This was our first idea. We created a login page component (LoginPage.vue) which uses our login button and defines three methods within it: googleLogin, facebookLogin, and emailLogin. These methods are called when a button triggers its click event.

<!-- LoginPage.vue -->

<template>
    <div class="login-page">
        <button-login label="Login with Google" :icon="googleIcon" @click="loginWithGoogle"/>
        <button-login label="Login with Facebook" :icon="facebookIcon" @click="loginWithFacebook"/>
        <button-login label="Login with email" :icon="emailIcon" @click="loginWithEmail"/>
    </div>
</template>

<script>
import ButtonLogin from './components/ButtonLogin.vue'

export default {
  name: 'LoginPage',
  components: {
    ButtonLogin
  },
  data () {
    return {
      googleIcon: {src: 'path/to/google-icon.png'},
      facebookIcon: {src: 'path/to/facebook-icon.png'},
      emailIcon: {src: 'path/to/email-icon.png'},
    }
  },
  methods: {
    loginWithGoogle () { /* google login logic */ },
    loginWithFacebook () { /* facebook login logic */ },
    loginWithEmail () { /* facebook login logic */ }
  }
}
</script>

This solution had nevertheless many drawbacks as our three login logics are contained in one single component. Consequently:

the LoginPage component is big and unclear
each connection methods is not reusable in other parts of the platform

By looking at the vue docs, we heard about Mixins. At first sight, we believed it was a great solution for the reusability of our login logics.

Second lead: Define each login logic in a different Mixin

A Mixin is an object containing component options (data, computed, methods...) that can be "merged" with a Vue component. During this process, the Mixin options are mixed with the component's options.

How does a Mixin look?

In our case, here was the GoogleLogin Mixin skeleton:

// mixins/GoogleLogin.js

export const GoogleLogin = {
  name: 'GoogleLogin',
  mounted () {
    /* init gapi client */
  },
  methods: {
    loginWithGoogle () {
      /* logic to open google popup, retrieve the authorization code and send it to backend */
    },
    onFailure () {
      /* logic when the sign in method fails */
    },
    onSuccess () {
      /* Logic sending the authorization code to my backend */
    }
  }
}

and our LoginPage:

// LoginPage.vue

<template>
  <div class="login-page">
    <button-login label="Login with Google" :icon="googleIcon" @click="loginWithGoogle"/>
    <button-login label="Login with Facebook" :icon="facebookIcon" @click="loginWithFacebook"/>
    <button-login label="Login with email" :icon="emailIcon" @click="loginWithEmail"/>
  </div>
</template>

<script>
import ButtonLogin from './components/ButtonLogin.vue'
import { GoogleLogin } from './mixins/GoogleLogin.ts'
import { FacebookLogin } from './mixins/FacebookLogin.ts'
import { EmailLogin } from './mixins/EmailLogin.ts'
export default {
  name: 'App',
  components: {
    ButtonLogin
  },
  mixins: [GoogleLogin, FacebookLogin, EmailLogin],
  data () {
    return {
      googleIcon: { src: 'path/to/google-icon.png', alt: 'google icon' },
      facebookIcon: { src: 'path/to/facebook-icon.png', alt: 'facebook icon' },
      emailIcon: { src: 'path/to/email-icon.png', alt: 'email icon' },
    }
  },
}
</script>

Mixins looked like an efficient way to encapsulate and share the different login logics through our application. Nevertheless, after a few tests, we experienced some strange behaviors while using Mixins:

implicit dependencies: the Mixin's methods and data are not defined within the LoginPage component. As such it is confusing to use "hypothetical" methods or data (and to not know where each method or data is defined).
naming collision: when merging the Mixins inside the page component, some of our methods were not called. They were overridden by methods with the same name but defined in another Mixin. As an example, if the FacebookLogin and GoogleLogin Mixins have an onFailure method, only the one defined in FacebookLogin will be called. Indeed, the FacebookLogin Mixin is imported after the GoogleLogin Mixin. In addition, if an onFailure method is also defined in the PageLogin component, then calling this.onFailure will trigger the onFailure method defined inside PageLogin. Consequently, none of GoogleLogin and FacebookLogin onFailure methods will be called (and this will happen without any warning or errors in the console).

We added to the list of our criteria the prevention of implicit dependencies and naming collision.

Knowing that, we were determined not to use Mixins. We decided to find a cleaner way to meet our objectives. It was at that moment we heard about the new composition API of Vue 3.

Third lead : Use the Vue 3 composition API

On September 18th, 2020, Vue 3 was officially released with a new great feature for reusability: the composition API. We won't get into the details of how the composition API works, but in a few words, this API is inspired by React Hooks*. It enables encapsulating component logic into small fragments of code reusable in different components.

* It has nevertheless a major difference: composition API hooks are called once and uses Vue's reactivity system while react hooks can be triggered multiple times during rendering

As an example, we could write this code fragment for google authentication logic:

// src/composables/useGoogleAuthentication.js

import { onMounted } from 'vue'

export default function useGoogleAuthentication() {
  const initGapiClient = async () => {
    /* initialization of google api client */
  }
  const loginWithGoogle = async () => {
    /* logic to open google popup, retrieve the authorization code and send it to backend */
  }

  onMounted(initGapiClient)

  return {
    loginWithGoogle
  }
}

Then, to retrieve the pieces of information required on our login page, the useGoogleAuthentication options can be merged into our page options (notably data, methods, and computed data but also lifecycle hooks). This merge is managed with the setup LoginPage component option.

// LoginPage.vue
<template>
  <div class="login-page">
    <button-login label="Login with Google" :icon="googleIcon" @click="loginWithGoogle"/>
  </div>
</template>

<script>
import ButtonLogin from './components/ButtonLogin.vue'
import useGoogleAuthentication from './composables/useGoogleAuthentication'


export default {
  name: 'App',
  components: {
    ButtonLogin
  },
  data () {
    return {
      googleIcon: { src: 'path/to/google-icon.png', alt: 'google icon' },
    }
  },
  setup() {
    const {loginWithGoogle} = useGoogleAuthentication();
    return {loginWithGoogle};
  },
}
</script>

* In order to prevent the loginWithGoogle method from triggering twice you will need to remove the custom click event in LoginButton.vue (@click="$emit('click') is no more necessary in Vue 3, the root element of a component will inherit the attribute)

As you can see, this solution prevents any implicit dependencies or naming collisions as you now need to clearly import the reusable part of your code. Reusing the authentication logic in different parts of the application is straight-forward.

This solution seems to be fitting our needs, right?

Unfortunately, we were working with a framework that was not yet supporting Vue 3. We had to find a workaround.

Good to know: As planned in the vue 3 roadmap, and especially because the support for IE may be stopped for Vue 3, some backport compatible features will be added to Vue 2.7. Among which the Composition API (the composition api plugin will be merged to Vue 2 Core). This change may enable Vue 2 users to benefit from the composition API features.

Solution: Use scoped slots

Looking a bit deeper in the Vue 2 documentation, we found the v-slot attribute (formerly named slot-scope). This attribute allows us to retrieve data or methods from a child component and pass it to the children of the component. Thus, for each connection method, we created a component including the connection generic logic and making the login method available to its children.

Here is an example with the GoogleAuth component. It encapsulates the google login logic and passes on loginWithGoogle method to its children.

// components/GoogleAuth.vue

<template>
  <div>
    <slot :loginWithGoogle="loginWithGoogle" />
  </div>
</template>

<script>
export default {
  data () {
    return {
      auth_code: ''
    }
  },
  mounted () {
    /* init gapi client */
  },
  methods: {
    loginWithGoogle () {
      /* logic to open google popup, retrieve the authorization code and send it to backend */
    },
    onFailure () {
      this.$emit('auth_error', { error: 'google error' });
    },
    onSuccess () {
      this.$emit('auth_success');
    },
  }
}
</script>

Then, the component GoogleAuth will wrap a button login and pass the login method to it. (Two other similar wrapping components for Facebook or email have to be created)

// LoginPage.vue

<template>
  <div class="login-page">
    <google-auth
      @auth_error="() => {/* manage authentication errors */}"
      @auth_success="() => {/* manage authentication success (e.g., navigate to another page) */}">
      <template v-slot="{loginWithGoogle}">
        <button-login label="Login with Google" :icon="googleIcon" @click="loginWithGoogle"/>
      </template>
    </google-auth>
  </div>
</template>

<script>
import ButtonLogin from './components/ButtonLogin.vue'
import GoogleAuth from './components/GoogleAuth.vue'
export default {
  name: 'App',
  components: {
    ButtonLogin,
    GoogleAuth
  },
  data () {
    return {
      googleIcon: { src: 'path/to/google-icon.png', alt: 'google icon' },
    }
  },
}
</script>

This solution answered our needs as it:

encapsulates the login logics in reusable components
splits the responsibility of the UX component (button) and the functional behavior (login logic)
doesn't create any uncontrolled side effects (each login method is scoped at the level of the button and not the page)

Conclusion

All the examples presented above allow us to compare different options to write and share component logic in Vue JS. In your project, you may choose a different option depending on the following criteria:

Criteria	JS Functions	Mixins	Scoped slots	Composition API
Encapsulate and reuse logic	✅	✅	✅	✅
Access to component options such as lifecycle hooks	❌	✅	✅	✅
No implicit or magic dependencies	✅	❌	✅	✅
No data collision	✅	❌	✅	✅
Readibility	✅	❌	✅ small component ❌ big component	✅
Where can shared logic be used ?	template & component	template & component	template	template & component
Available in Vue 2	✅	✅	✅	〜
Available in Vue 3	✅	✅	✅	✅

In a nutshell, for medium or big projects in Vue3, I would highly recommend using the composition API as it enables to write readable and reusable component logic anywhere in the code (including outside of components). For Vue 2, use scoped slots (for now) as it is a pretty straight-forward way to share component logic in Vue 2 without the drawbacks of mixins. It is even the best option for code reuse when dealing with a reusable component that provides a customizable UI in both Vue 2 and Vue 3.

A Newbie's Guide to React Query

Thu, 03 Jun 2021 00:00:00 GMT

Let's say you're building an application that relies heavily on displaying and interacting with database information (super rare situation as a dev, I know). You're also pretty good at React and Redux, so you start building your app with those libraries.

Next, you make an API request for a dataset and put it in your Redux store. Then you render the Redux store data.

Yep, I'm following. So far so good

Next comes manipulation of that data! You build out some actions that handle adding, editing, and deleting data.

Yeah, that's pretty standard!

Then, you trigger another GET of the data. React re-renders for you and you're now displaying the updated data set!

But what do you do if the API you're working with is slow? Getting that updated data back could lead to a delay in rendering the latest data, leading to a subpar user experience.

Hmm... guess I could update the store data?

True, but what then happens if the API call was not successful? Now the data in the Redux store is out of sync with the API data. How should we handle this situation? Keep track of the previous state to fall back to? How would we write up the logic for that?

Where is your god now?

Well, it seems we've stumbled upon a common problem — so common it feels like there should be a simple solution. Luckily for us, some great solutions are out there. Let's dive into my favorite one!

What is React Query and how does it solve our problem?

The React Query site claims the following: "Performant and powerful data synchronization for React. Fetch, cache and update data in your React and React Native applications all without touching any 'global state'."

Sounds pretty good, right? As we've discussed earlier, holding data in our global state can cause a host of issues. As React Query's author Tanner Linsley says, "It's time to break up with your global state!"

Let's take a look at some sample code:

Now this is where the magic happens: the first argument in useQuery, "todos", represents the query key.

Okay, but it just looks like a random string to me..

The query key is core to React Query — it will automatically cache query data based on these keys. Thus, you should always write and use query keys that are unique to the query's data. While this example's query key is a simple string, the query key can also be an array that contains strings, variables, and even objects!

Nice! Now we're good to go, right?

Almost! Now we're using React Query and have access to our data. And as we just learned, caching is already built-in! But are we already using it? Do we have to enable it somehow?

React Query Cache Basics

As it turns out, the code we saw above doesn't use React Query's cache just yet. First, we have to assign a value to the staleTime property.

...huh?

Think of staleTime as a (millisecond) countdown timer. When the timer expires, React Query then marks the query data it has cached as "stale." As a result, the query data is refetched in the background at the next available opportunity (see the docs for a list of situations that can trigger this). You can even set this value to Inifinity, for which it will never be stale!

Gotcha. So, why again did we need to set it in the first place?

If we peek at this page in the docs, we find that React Query is configured with "aggressive but sane defaults," one of which is the default setting of 0 for staleTime. That's pretty aggressive if you ask me!

Alright, now show me some code!

We've now assigned the value of one minute to staleTime! This way, the next time a query is fired with the query key 'todos' within the next minute, it will return the cached data. Woohoo!

Nice! But what if the data changes within that minute though?

That's right! Since our data fetching has this staleTime property, we may not see my new data for at most another minute — or even longer if we set a longer staleTime! So what do we do now?

As you've probably guessed, the clever folks who write and maintain React Query have got us covered. We have an invalidateQueries method on the QueryClient that we can use that invalidates query data by marking it stale. However, we can do even better!

React Query Mutations

Enter the almighty React Query mutation! Mutations give us a way to create, update, or delete data — and allow a lot of flexibility with how the query data is affected as a result.

Sounds great, where do I sign up?

We can get started using React Query mutations with the useMutation hook. The useMutation hook provides many useful options but we'll be focusing on a few: onMutate, onError, and onSettled.

onMutate is fired just before the actual data manipulation function — updateTodo, in the example below.

Wait, what's happening here? setQueryData?

Good eye! We've also made use of yet another handy React Query feature here: Optimistic Updates. Remember how we'd have a delay in rendering the latest data if our API response came back too slow?

Yeah...

We solved that here simply by calling the setQueryData method! The first argument is the query key, and the second argument is a function where we have access to the old query data. The function expects you to return new data, so we can return the old data with our newToDo appended to it.

Great, so we're done now!!

Not so fast! What we've done so far is improve the rendering of new data when things go well and we are on the happy path. But, what happens if our create, update, or delete calls don't go as planned?

Again, thanks to React Query, we have the ability to easily snapshot the old data and fall back to it! The first step is to "snapshot" it by saving it in a variable, then returning it, like so:

Next, we'll simply use that snapshot in our onError function and call setQueryData with it!

Great, now we're done! ...right?

Yep! Just kidding, we're not done yet! There's one more thing we should do: remember the invalidateQueries method we briefly mentioned before? This marks the data as stale and tells React Query that the query data tied to this particular query key should be refetched. The onSettled function gets fired at the end of a mutation whether the result was successful or not. This seems like a good place to mark the data as stale:

Here's what we have now. This will handle updating the todo item, roll back in case of an error, and mark the dataset as stale to trigger a refetch. Pretty neat if you ask me!

This piece of code does all that? Awesome!

Key takeaways

When implementing React Query, remember to always use the same query key for its matching data set
If you want to make use of caching, be sure to set a staleTime with your query
When adding new, editing, or deleting data, use a mutation to help you handle synchronization and fallback logic

If your application has a need for data synchronization, why not switch to using React Query?

Concluding thoughts

React Query gives us the luxury of not having to think too hard about how to keep our data in the front end up to date with our database information. Global state management is a great tool, but a lot of thought and careful planning has to go into handling edge cases and logic if things stray from the happy path. React Query makes handling all of this much easier, and even comes prepackaged with many powerful features like caching!

Looking to get a bit more in depth? Let me know if you’d be interested in a part 2 in the comments below!

How to Prevent Springboot Crashes after a Checkout

Sat, 15 May 2021 00:00:00 GMT

At Theodo, I'm working on a project where we need to start 5 APIs simultaneously where 40 developers are continuously pushing new lines of codes and our stack is relying on Spring Boot. Unfortunately one of our favorite action during our daily dev routine is just crazily complicated: 10% of the time, checking out from a branch to another one break the build of our APIs. It's a real pain point and we lose hours in debugging for something that is not a bug!

My problem is: How to checkout smoothly without worrying about API compilation being different between branches?

TL;DR

This script composes the post-checkout hook to automate how to cleanly restart your Spring Boot APIs:

#!/usr/bin/env bash
IFS=$'\n\t'

WORKDIR=$PWD
PREVIOUS_HEAD=$1
NEW_HEAD=$2

APIS="
users-api
payment-api
mailing-api
"
for nameApi in $APIS;
do
  cd ./$nameApi/src/

  GIT_DIFF=$(git diff $PREVIOUS_HEAD...$NEW_HEAD --relative)
  cd $WORKDIR
  if [[ ! -z "$GIT_DIFF" ]]; then
    if [[ "$(docker-compose ps $nameApi | grep Up)" ]]; then
      rm -rf ./${nameApi}/target
      docker-compose restart $nameApi
    else
      rm -rf ./${nameApi}/target
    fi
  fi
done

What happens when you checkout?

Let's take a simple example; I have to review the code of feature_branch_2, I thus have to jump often from feature_branch_1 to feature_branch_2.

The diagram above illustrate the differences between the 2 branches :

Model 1 doesn't exist in the second branch
Model 2 has some changes between the two branches

During the checkout, the change of branch can be brutal for Spring Boot and the automatic reloading does not manage to notice all the changes. Some of them may be missing from the compiled files and the server may crash.

My team was using a trick to circumvent the problem, it was performed through 3 painful operations :

Stop the server
Clean the builded target files
Rebuild the docker image and start the server (rebuilding the docker image included a target files build)

After some tests, we optimized the first and third step and only had to do the following :

Clean the builded target files
Restart the server

We reduced the number of steps but who wants to do 2 (if not 3) commands like this at each checkout?

How to automate this routine?

The best way I chose to automate was the post-checkout action that is native in Git versioning system. The idea is simple, after each checkout, a bash script is automatically executed based on the difference between the start and end branches of the checkout. This script is the following:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

WORKDIR=$PWD
PREVIOUS_HEAD=$1
NEW_HEAD=$2

APIS="
users-api
payment-api
mailing-api
"
for nameApi in $APIS;
do
  cd ./$nameApi/src/

  GIT_DIFF=$(git diff $PREVIOUS_HEAD...$NEW_HEAD --relative)
  cd $WORKDIR
  if [[ ! -z "$GIT_DIFF" ]]; then
    if [[ "$(docker-compose ps $nameApi | grep Up)" ]]; then
      rm -rf ./${nameApi}/target
      docker-compose restart $nameApi
    else
      rm -rf ./${nameApi}/target
    fi
  fi
done

The idea is to execute the clean and the restart for each API of my application, only if the sources' files from one branch to another contain differences.

The clean allows for a clean start of the API whether it is up or down. Additionally, I restart the API if it was already up before the checkout!

To make this script working on your project, you just need to copy/paste it in a file named .git/hooks/post-checkout.

What about the first lines?

I got into the habit of starting my bash scripts with the set -euo pipefail option allowing it to stop at the correct line if there is an error. It's easier to debug if required.

Also, the variable IFS=$'\n\t' allows the terminal to set the internal field separator.

To go further

The major problem with putting your script in the .git folder is that you can't make the most of Git to version and share it with your whole team. Indeed, this folder is often ignored in the repo and has no history. In fact someone else in the team brilliantly discovered the Husky library. With this library, you can put your script in a versioned folder (let's call it scripts at the root of your project) and configure Husky to call it whenever you want (let's say at the post-checkout event obviously). You can then have this type of configuration in your package.json:

{
  ...,
  "husky": {
    "hooks": {
      "post-checkout": "./scripts/post-checkout-hook.sh $HUSKY_GIT_PARAMS"
    }
  },
  ...
}

And if you want more with the Git hooks, there are plenty of other applications. For example, you can use the pre-commit hook to run the linter and format the code before sharing it with others! How many times had we changed a few things in a file and the saving started formatting everything because the previous developer who worked on the file didn't set up the IDE's linter...

You also can deal with other hooks proposed by Git, Reynald Mandel made a great example of post-merge hook use.

So what?

To talk about the limits, the script is executed each time we checkout and if the git diff resulting in a huge number of files, the execution can use a lot of memory. Just not that it's limited by the architecture of the script, we actually don't check each file, we just check if there is a difference or not.

Finally, with this hack, I reduced the frustration of the developers in the team, and we gained 40 hours of work per week at the scale of all the team (40 developers). This also represents the sobering saving of a developer on our client's bill!

Spice up Your Website with a DocuSign Electronic Signature Embedded Tool

Fri, 14 May 2021 00:00:00 GMT

Introduction

Integrating electronic signing in a website is not always so easy. If you ever get tired of spamming users who already have an overflowing mailbox with DocuSign emails, then you have landed on the right article. Let me show you how to replace them with an elegant embedded tool that will boost your platform.

This article addresses the following problems:

Finding a simple way to make people sign a disclaimer before accessing the content of a certain page;
Using DocuSign in a modal;
Embedding everything on your website.

Why this one-page architecture?

The goal of this architecture is to make sure users have to go through the disclaimer check before accessing the page. If you've got two different pages for that purpose, any decent attacker could bypass it:

Yup, it's almost as simple as that...

Why not using DocuSign emails?

When developing a feature using DocuSign, you will probably have to pick one of these two options:

Embed DocuSign signing;
Send DocuSign emails (which seems to induce less coding thus sounds more attractive).

After going through a lot of pain playing around and debugging unsent emails, I figured that not having to worry about unsent, unread, or spam-categorized emails was a fair point for the first option. The second reason is simple: it's easier to track issues, as everything - or almost - is embedded within your website. Last but not least, it came up that having a linear path would boost the UX by removing external dependencies.

Not paramount but worth mentioning, it also makes it easier to perform E2E tests on this feature.

In overall, this solution was:

Simple enough to be quickly implemented;
Customizable enough to fit all my different cases;

Installation

Simply run pip install docusign-esign to get started.

Let's break it down

We will assume in this article that the user has not signed the disclaimer yet.

Server-side

As shown in the schema above, the backend falls in two steps :

Defining the DocuSign envelope

The backend receives user information (email, name, ...) from the frontend. Then, we use the DocuSign Python Client to define our DocuSign envelope, i.e which document(s) will be part of the envelope, where they should be signed, who should sign them, and so on. This results in an envelope definition that we send to DocuSign to create it.

Getting the signing URL

DocuSign returns an envelope ID corresponding to the created envelope. It is used to start the signing ceremony, meaning that it is sent to DocuSign to generate a signing URL. Ultimately, this signing URL is sent back to the frontend.

Bear in mind that we can pick at this stage the redirection URL, i.e to which URL the user will be redirected once the signing ceremony is finished (necessary to understand our trick!).

Macro implementation (view)

Following this logic, the implemented view should create a DocuSign envelope and return its signing url to whomever calls the related endpoint.

docusign_service = DocuSignClient()

class DocuSignViewSet(APIView):
    def get(self):
        envelope_id = docusign_service.call_docusign_create_envelope(
            full_name=full_name,
            email=email,
            contact_crm_id=contact_id,
            document_path=document_path,
            document_name=document_name,
        )
        signing_url = docusign_service.start_signing_ceremony(
            envelope_id=envelope_id,
            full_name=full_name,
            email=email,
            contact_crm_id=contact_id,
            redirection_route=<CUSTOM_REDIRECTION_ROUTE>,
        )

        return Response(signing_url)

Micro implementation (DocuSignClient class)

Click here to discover details about its implementation

To make it easier, let's create a DocuSign class with all necessary methods. This will let us keep the same valid token during the whole process and renew it if needed.

import jwt
from docusign_esign import *

class DocuSignClient:
     token = None

Now, to communicate with the DocuSign API, what we first need is a way to get a valid EnvelopesApi object, i.e that has a valid token and correct parameters (pathname and headers), so let's add these two methods:

     def get_client(self):
         api_client = ApiClient()
         api_client.host = <MY_DOCUSIGN_API_BASE_PATH>
         api_client.set_default_header("Authorization", f"Bearer {self.token}")

         return api_client

     def get_envelope_api(self):
         api_client = self.get_client()
         envelope_api = EnvelopesApi(api_client)

         return envelope_api

However, because we want the token to be valid and to automatically renew if not, I added the following pre-check:

     def get_client(self):
         if not self.is_token_valid():
             self.token = self.renew_token()

        # ...

As you will see below, this check is essentially a call to the DocuSign API using the current token to verify whether it is valid. If it is expired or invalid (401 status code), we renew it. We use a single login for all users in the platform. Therefore, we use the JWT Grant authentication as suggested in the documentation.

     def is_token_valid(self, method="GET"):
         headers = {"Authorization": f"Bearer {self.token}"}
         response = requests.request(
             method, f"{<MY_DOCUSIGN_API_BASE_PATH>}/v2.1/accounts/", headers=headers
         )

         if response.status_code != 401:
             return True

         return False

     def renew_token(self):
         current_datetime = datetime.datetime.now()
         jwt_token = jwt.encode(
             {
                 "iss": <MY_DOCUSIGN_INTEGRATION_KEY>,
                 "sub": <MY_DOCUSIGN_API_USERNAME>,
                 "name": <NAME>,
                 "aud": "account-d.docusign.com",
                 "iat": current_datetime,
                 "exp": current_datetime + datetime.timedelta(minutes=30), # validity duration of 30 minutes
                 "scope": "signature impersonation",
             },
             key=<MY_DOCUSIGN_PRIVATE_KEY>.encode(),
             algorithm="RS256",
         )

         headers = {"Content-Type": "application/x-www-form-urlencoded"}
         url = f"{<MY_DOCUSIGN_BASE_URL>}/oauth/token"
         payload = {
             "grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
             "assertion": jwt_token,
         }
         response = requests.request("POST", url, headers=headers, data=payload)

        return response.json()["access_token"]

Believe me, all these methods will simplify your life. You'll only have to call get_envelope_api() to get a functional EnvelopeApi with a valid token.

Once done, we can then add methods to create a DocuSign envelope, which is roughly a container for documents that you send to a recipient for signing.

     def make_docusign_envelope(
         self,
         signer_full_name,
         signer_email,
         contact_id,
         document_path,
         document_name,
         anchor_y_offset="0",
         anchor_x_offset="0",
     ):
         with open(document_path, "rb") as file:
             content_bytes = file.read()
             base64_file_content = base64.b64encode(content_bytes).decode("ascii")

         document = Document(
             document_base64=base64_file_content,
             name=document_name,
             file_extension="pdf",
             document_id=1,
         )

         signer = Signer(
             email=signer_email,
             name=signer_full_name,
             recipient_id="1",
             routing_order="1",
             client_user_id=contact_id,
         )

         sign_here = SignHere(
             anchor_string="/sn1/", # the anchor string to indicate where to sign on the PDF
             anchor_units="pixels",
             anchor_y_offset=anchor_y_offset,
             anchor_x_offset=anchor_x_offset,
         )

         signer.tabs = Tabs(sign_here_tabs=[sign_here])

         envelope_definition = EnvelopeDefinition(
             email_subject="Please sign this document",
             documents=[document],
             recipients=Recipients(signers=[signer]),
             status="sent",
         )

         return envelope_definition

The make_docusign_envelope() method allows you to design a fully customizable envelope, by defining all components such as the document content, signers, locations where a signature has to be attached... You can then use it as a template like so:

     def call_docusign_create_envelope(
         self, full_name, email, contact_crm_id, document_path, document_name,
     ):
         envelope_api = self.get_envelope_api()
         envelope_definition = self.make_docusign_envelope(
             full_name, email, contact_crm_id, document_path, document_name
         )

         results = envelope_api.create_envelope(
             <MY_DOCUSIGN_API_ACCOUNT_ID>, envelope_definition=envelope_definition,
         )

         envelope_id = results.envelope_id

         return envelope_id

Indeed, call_docusign_create_envelope will make use of the previous method to create an envelope based on this template and returns an envelope ID that can be used to start the signing process (a.k.a ceremony).

    def build_docusign_redirection_url(self, redirection_route, options):
         return <MY_FRONTEND_URL> + redirection_route + options

     def start_signing_ceremony(
         self,
         envelope_id,
         full_name,
         email,
         contact_crm_id,
         redirection_route,
         authentication_method="none",
     ):
         envelope_api = self.get_envelope_api()
         return_url = self.build_docusign_redirection_url(redirection_route, envelope_id)
         recipient_view_request = RecipientViewRequest(
             authentication_method=authentication_method,
             client_user_id=contact_crm_id,
             recipient_id="1",
             return_url=return_url,
             user_name=full_name,
             email=email,
         )

         results = envelope_api.create_recipient_view(
             <MY_DOCUSIGN_API_ACCOUNT_ID>,
             envelope_id,
             recipient_view_request=recipient_view_request,
         )

         return results.url

Note that the custom redirection url built by build_docusign_redirection_url is the one that will be used frontend-wise once the signing ceremony is completed, i.e when the user clicks on the "Finish" button. Completing the ceremony will also update the envelope status in DocuSign to completed.

Client-side

Add DocuSign in an iframe

When the user accesses the restricted page, the backend is called to get a signing URL. Our modal contains an iframe that will use this URL to embed DocuSign content, meaning DocuSign envelope signing portal with all documents which have to be checked.

export const DocuSignModal: React.FC = () => {
  const { signingUrl } = getDocuSignSigningUrl(); // calls the previously created DocuSignViewSet

  return (
    <Modal>
      <iframe src={signingUrl} />
    </Modal>
  );
};

Use the redirection link to update the parent component

As seen previously, once the user signed all necessary documents and clicked on DocuSign "Finish" button, the URL provided when the signing ceremony started is used to redirect the user. The issue here is that the redirection will occur within the iframe since it is a different browsing context. And of course, we do not want that.

And now the trick!

Remember what we said about having free hands on the redirection URL? Well, we used this feature to build a URL such that it points to a page containing a shadow component and contains the corresponding envelope id. It will look like so: https://my-website.com/my-redirection-page/[ENVELOPE_ID]. Once rendered, this shadow component will post a message to the parent component (i.e the modal). This message contains the envelopeId of the created envelope, which can be found in the URL.

export const MyShadowComponent: React.FC = () => {
  const router = useRouter();

  useEffect(() => {
    window.top.postMessage(
      JSON.stringify({
        success: true,
        envelopeId: String(router.query.envelopeId),
      }),
      window.location.origin
    );
  });

  return null;
};

This message sent from the iframe will be listened to by the parent component (the page itself), which can then perform actions such as closing the modal, refreshing, routing to another page...

// event.origin and window.location.origin have to be compared for security reason
const isSameOrigin = (event: MessageEvent) =>
  event.origin === window.location.origin;

useEffect(() => {
  const handler = (event: MessageEvent) => {
    if (event.data && isSameOrigin(event)) {
      const { success, envelopeId } = JSON.parse(event.data);
      if (success) {
        // Do something such as closing the modal, saving the envelopeId, ...
      }
    }
  };
  window.addEventListener("message", handler);

  return () => window.removeEventListener("message", handler);
}, []);

You're done!

Conclusion

I hope you found interest in this article which highlights a simple way to embed DocuSign in your website. There are still many ways to improve and customize this flow, with additional features such as:

Saving in the backend users who have already signed so that they are not asked to sign again later ;
Tracking documents versions.

Hope that was useful, cheers!

Useful links

How to develop on Shopify in 2021

Wed, 28 Apr 2021 00:00:00 GMT

As a software developer, I often hear that No-Code is the future of app development. That in a few years, automatically generated code will replace all the frameworks that we currently have. With my young experience in this field I'm quite sceptical, because I still believe that to truly answer a problem or a need you will always need something that can embrace it completely.

However, I was quite amazed by my last and first experience using one of the major tools in the No-Code business: Shopify.

Shopify became in the last few years the reference for online stores, and is now powering more than 18% of all eCommerce stores (according to BuiltWith). Their success can mostly be explained by the convenience of their product: by only using the platform, you can build a highly customized store, ready to use, in only 2 hours.

However, if you have very specific needs regarding your online store, you will quickly need a developer to be able to deep dive into Shopify's technical documentation. Therefore, I decided to write this article to summarize the different things that I learnt using Shopify as a developer, and how I used them in my project.

Here are the different topics that will be covered:

Front framework: Liquid
Embed a React App inside Liquid's engine
Shopify's API - Usage and Limitation
Development Workflow: Automatize your deployment with CircleCI, Github and ThemeKit

Front framework: Liquid

Even though Shopify has the specificity to be used as a headless ecommerce solution (using their API), the native behavior of Shopify is to use their embedded front system (Wordpress-like).

And for our project, we decided to stick with the classic front system as it allowed us to have an instant-ready website, giving the possibility to our client to test different configurations.

So, we had to deal with their own programming language: Liquid.

Liquid is an open-source template language written in Ruby. As many template languages (like Jinja (Python) or Twig (PHP)), it makes a bridge between data and HTML, in which you can add logic (if, while loops...). In Shopify's usage of Liquid, the data is the different entities related to your store (products, cart, collections...).

Before launching yourself into any Shopify development, I highly recommend you to read this introduction to the Liquid language, written by Keir Whitaker, a former Shopify Partner. It describes in an exhaustive way how Liquid is built and how Shopify use it for their front framework.

It also mentions one of the main downsides that we encountered during our project: Liquid doesn't have any concept of state, which can really be a pain point if you are willing to make your website a little bit interactive, by creating a real user experience. In our case, we wanted to create an interactive quiz, in which our users could answer a few questions about themselves, and push to them the best matching products.

In order to develop our feature inside Shopify's front engine, we decided to use React's framework, embedded inside a liquid template.

Embed a React App inside Liquid's engine

Our challenge here was to be able to use the ease of front development with React without creating a separated hosted webapp. Our idea was to develop the React App in the same repository, and then build the static files (as for production) and add them to the assets of a Shopify theme.

Here was our deployment script:

#!/bin/bash

APP_BUILD_FILE=build/index.html
THEME_FILE=theme_live/sections/quiz.liquid

echo 'Adapting html file for shopify asset urls'
sed -E 's/(<script src=")\/static\/js\/([a-zA-Z0-9._-]*)("><\/script>)/\1{{'"'"'\2'"'"' | asset_url }}\3/g' $APP_BUILD_FILE > $THEME_FILE

echo 'Replacing built javascript files in theme'
rm -f theme_live/assets/quiz*
cp build/static/js/* theme_live/assets/

Once a React App is built, the entry file is the index.html that contains all the script tags pointing to the js files of your app, divided in different chunks. The first command of our script catches all the script tags contained in the index.html file and then replaces them by a liquid syntax.

From:

<div id="root"></div>
<script src="/static/js/quiz_runtime-main.8923d629.js"></script>
<script src="/static/js/quiz_2.20b50bf0.chunk.js"></script>
<script src="/static/js/quiz_main.f9693573.chunk.js"></script>

We now have:

<div id="root"></div>
<script src="{{'quiz_runtime-main.8923d629.js' | asset_url }}"></script>
<script src="{{'quiz_2.b689c532.chunk.js' | asset_url }}"></script>
<script src="{{'quiz_main.d6725a08.chunk.js' | asset_url }}"></script>

The second command then removes the previous react JS files and replaces them with the new assets, in our quiz section.

However, this creates a separate app that can't interact with the liquid framework. Therefore, we needed to analyze how we could retrieve data from our Shopify store and display it to the user. Hopefully, Shopify has a great API, which allows you to interact with a lot of different elements of your stores.

Shopify's API - Usage and Limitation

This is where Shopify becomes very interesting. As said before, using the built-in Liquid framework allows you to create a ready-to-use store in only 2 hours. However, you'll sometimes find yourself in need to interact with your store data outside of your storefront, or maybe you want to only use Shopify as a backend, and make a completely custom frontend ,using gatsby or NextJS.

This gives you the possibility to move from a Liquid frontend to a much more customizable interface, by using Shopify's back API.

Shopify has 2 APIs available:

The Admin API (GraphQL and REST)
The Storefront API (GraphQL only)

Shopify's Admin API

The Admin API is the main API that allows you to interact with ALL the elements of your store. You can use it to follow your orders, products, payments... Well, pretty much everything. This API is mainly used by shopify's app developers, to extend the behavior of the basic Shopify back office.

In my case, I used it to create our product recommendation algorithm. As said before, our challenge was to create a quiz in order to advise our customers on a very specific market. We wanted to know:

The characteristics of the products they were looking for
Their family composition
What type of products they were looking for

Once the answers were given, our app then called the Admin API, in order to retrieve the best matching products. However, this is where I discovered the API's rate limitations. Indeed, our main goal was to consider all the selected product categories by our user (that could go up to 15 categories - the aim of our quiz was to create big buckets of products), and to propose in each category the best product.

We went for the GraphQL API because it allowed us to:

Make a single call with different product categories
Get only the data that we needed

However, GraphQL's API usage is limited by the cost of your query. This means that you cannot make a huge query in order to retrieve all your data at once. With classic Shopify, your query can have a maximum cost of 1000.

As we didn't have a lot of products per category, we limited the number of products per category to retrieve per 20, and limited to the maximum the data that we wanted to retrieve:

productType1: products(first: 20, query: "product_type:'productType1'") {
    edges {
        node {
            id
            productType
            tags
            priceRangeV2 {
                minVariantPrice {
                    amount
                }
            }
        }
    }
}
productType2: products(first: 20, query: "product_type:'productType2'") {
    edges {
        node {
            id
            productType
            tags
            priceRangeV2 {
                minVariantPrice {
                    amount
                }
            }
        }
    }
}

Once we retrieved the best matching products, we then sent back only the selected productIds, to give the possibility to our front to call the storefront API.

Shopify's Storefront API

Storefront's API is a way to expose your store's data to external apps (public or private), unauthenticated. The perfect usage of this API is when you want to use Shopify only as a merchant platform, and create your custom front platform (Shopify as a headless ecommerce solution).

It is important to remember that it's an unauthenticated API, meaning that once the Storefront's API is enabled, everybody can access the data that you exposed. However, you can control the data that you expose (you can only display the product details for example).

We used the storefront API for our store directly in our React App, in order to display the different products, and call the checkout to add the different items to the user's cart. We got inspired from the different adaptation of the buy-sdk adapters that you can find on Shopify's Github.

Development Workflow: Automatize your deployment with CircleCI, Github and ThemeKit

Last but not least, I will present the development workflow that we set up on our project.

Shopify has developed a command-line tool called Theme Kit, enabling you to interact with your shopify store and deploy code instantly.

Once Theme Kit is installed on your computer/server, you need to configure it inside your app folder, following this tutorial.

Theme Kit allows you to (in particular):

Preview your code as you code it on your shopify store theme using the theme watch command
Deploy code to one of your shop theme using theme deploy --env=development or any other name that you would have defined in your config.yml file.

development:
  password: ${SHOPIFY_PASSWORD}
  theme_id: ${SHOPIFY_DEV_THEME_ID}
  store: store-name.myshopify.com
  timeout: 1m0s
production:
  password: ${SHOPIFY_PASSWORD}
  theme_id: ${SHOPIFY_THEME_ID}
  store: store-name.myshopify.com
  timeout: 1m0s

Each developer had their own development theme (you can have up to 20 themes on one shopify store), and their was one "live theme" which was the production theme. The different useful variables were stored in a variables file, that Shopify's engine can interpolate. For further information, you can check Theme Kit configuration documentation.

Here was our developer workflow:

Get the latest code from the master branch using git
Push my code to my Shopify development theme using theme deploy --env=development
Make modification while using theme watch (so that you could see directly the changes on your Shopify theme)
Push your code, review it on github
Merge on master.
Once the code was merged, we had a CI/CD script that:
- Built the React App and moved the built files into the assets of our shopify theme
- Deploy the code to Shopify using Theme Kit

The last step was done using the following script in your CI:

echo 'Deploying runinng cd theme_live && theme deploy --allow-live'
cd theme_live && theme deploy --allow-live --env=production

The Shopify production Theme ID and user credentials were injected inside our CI configuration, to be able to deploy to the production environnement.

Conclusion

As you can see, on top of being a very easy-to-use product for non-tech people, Shopify is also a real state-of-art product. They thought of a simple workflow to automate your deployment, you can easily access your store using a highly documented API.

They created a real game changer when it comes to build an ecommerce website: you can start with a prototype using the classic Liquid templating product, then if you want to fully customize your storefront, you can change to a headless CMS version of Shopify using their API, and then you can even scale up by using their premium version Shopify Plus, allowing to manage different stores in different country.

And one thing that is important to notice: once your store is created, it's already usable. Shopify has their own secured payment platform, PIM management, and many other features that are required in every ecommerce store, which you don't need to develop anymore. And icing on the cake, by using Liquid's framework, your SEO will be optimized and fully manageable from your store.

I highly recommend each developer to take the time (with Shopify's 14 days trial) to build their own Shopify store, as it's a great skill to have in a world where the shopping experience is becoming more and more digitized.

How to Check Your Website on Mobile in the Local Environment

Wed, 28 Apr 2021 00:00:00 GMT

At Theodo, we develop solutions for both mobile and desktop. Most of the time we use the chrome device toolbar to see the visual rendering of the website when we develop in the local environment. Then when the code is deployed, we also check in the development/staging environment that everything works correctly.

However, sometimes, this is not enough. For example, this is not applicable to debug a behavior that occurs only on mobile devices. The obvious solution: finding a way to check the website on mobile, in the local environment. There are several ways to do that and each has its perks and drawbacks. In this article, I will present you with those solutions to help you find the one that suits you best.

For you to find the best solution in your use case, I will evaluate each solution with the following criteria:

📱 ‍Provides an experience similar to the end-user interaction on mobile
📋 Easy access to logs
🔧 Easy to set up

Chrome device toolbar

Before you start, make sure that the feature you want to check cannot be checked with the Chrome device toolbar.

Chrome device toolbar is accessible by a single click on the mobile icon at the top of the Chrome inspector

It does what you expect and maybe more…

Chrome device toolbar is an obvious solution, known by most developers, that allows you to test responsiveness on a large variety of devices. But responsiveness is not the only thing you can test with the chrome device toolbar. All the available options are presented in great detail in the chrome documentation. If you need an incentive to go see for yourself, be aware that the device toolbar allows you to override geolocation, throttle your CPU and/or internet speed, display media queries… A less known feature of the Chrome device toolbar is that it properly sets the user-agent according to the device you choose. So if you implement a different behavior based on the userAgent, you will be able to see it.

But Chrome device toolbar is not an emulator

However, as complete as it is, the chrome device toolbar is still limited. Here is a quick list of things you may need, and that is still unavailable with this tool:

The fluidity of scrolling and swiping. One of the main differences between desktop and mobile is the way the user interacts with the device. And this is a big drawback of the Chrome device toolbar. Your scroll could be lagging like hell on a mobile device, you would not be able to see it.
Opening/closing of keyboard and focus. Don't forget that the Chrome device toolbar remains more or less a desktop website visualizer that can take the shape of a mobile device. It won't display the behaviors that are particular to mobile such as the opening of a keyboard.
Running code that is specific to an operating system. A video player designed to run on IOS won't run in the chrome toolbar even if you selected an iPhone.

Solution evaluation

📱 Close to end-user experience: Mediocre

❌ Chrome device toolbar is not an emulator.

📋 Access to logs: Excellent

✅ No need to change development flow to retrieve logs. Everything you need is directly available in the browser.

🔧 Easy to set up: Excellent

✅ A click on a button in the inspector.

Chrome device toolbar is enough for basic responsiveness but what if we need to access our website / debug on a real mobile device ?

Using your local network to access your website in the local environment

There is a quick solution that doesn't require any third-party software and that allows you to connect to your website from your mobile on your local network. All you need is a mobile device and a computer connected to the same network, i.e. a wifi network, or a shared mobile internet connection.

The principles

As you probably already know, when setting up the services for your application, you give them an address which on your local environment is localhost + port number. You can make your website accessible to all the users on the local network by exposing it to your address on this network.

The reason for this is rooted in the difference between localhost and your local IP address. Localhost is an internal address in your computer used by the loopback interface, a virtualized network interface. Localhost is only used inside your computer. On the other end, the local IP address is an address given by your router for your computer to communicate with other devices on the local network.

How to set it up

Identify your local Ip address. On Windows, you can open the terminal and execute ipconfig. On Mac and Linux, the command to execute is ifconfig. But maybe you won't even need to do that. For example, when starting a React App, the local IP address is displayed in the console.

You can see your local IP address when launching a React App as "On Your Network"

If you only have a frontend, you are already good to go. Just enter the local IP address in the URL bar of your browser with the right port, and you will see your website.
If you have a backend, you should replace your backend adress(es) called by the frontend from

http://localhost:<your_backend_port>

http://<your_local_ip_address>:<your_backend_port>

Your website is now accessible at the following address on both your desktop device and to any device connected to the local network:

http://<your_local_ip_address>:<your_front_port>

A nice addition to this is to add a script such as the one below in your package.json to automatically fetch your local IP address and update your frontend/backend URLs in your environment files:

#grepping the local ip address
LOCAL_IP=$(ifconfig
| grep -Eo 'inet (addr:)?([0-9]*\\.){3}[0-9]*'
| grep -Eo '([0-9]*\\.){3}[0-9]*'
| grep -v '127.0.0.1');
# using a .env.mobile file as a template
# and moving the output into .env file
sed \"s,frontend_url,$LOCAL_IP,g\" .env.mobile > .env;
# starting the react app with the updated file
react_app start

Make CORS allowed host changes if needed. You will only have to do the same replacement as the previous point for your front-end address.

How to see the logs

Now that you are accessing the website in your local environment on a mobile device, the next step would be to see the logs of your device so you can develop and debug your application correctly.

MacOS ⇔ IOS. For this, you can use the Safari development tools. Connect your device to your computer using a USB cable and when asked if you trust the computer, answer yes. Open Safari and click on the 'Develop' tab

You can access your mobile logs by opening the Safari Browser. Select first the 'Develop' tab then the 'name of your device' and finally the 'name of the page'
Mac/Windows/Linux ⇔ Android: Google is providing a tool to access logs by connecting your mobile to your computer using a USB cable .
Windows/Linux ⇔ IOS: It seems there is no obvious solution for this currently.

A versatile solution that covers (almost) any use case

What is great about this solution is that it allows you to test directly on your mobile device and also that it works not only for a frontend but also for a "complete website" with a/multiple backend and a database as well.

But:

This solution requires updating all the addresses throughout your application. However, this can be easily solved by adding a script to your package.json
Some third-party API requires HTTPS and won't work with this setup.
Your router may change your local IP address from time to time.

Solution evaluation

📱 Close to end-user experience: Good

✅ You are debugging directly on your mobile device.
❌ You can only use this method on the devices you own, so you may overlook some edge cases on other devices.
❌ You also may encounter errors if HTTPS is mandatory for some APIs you use. In that case, see my next solution.

📋 Access to log: Very good - No Logs

✅ The same quality of logs as when you are developing for computer users...
❌ ...Except for the IOS + Windows/Linux duo.

🔧 Easy to set up: Very good

✅ Relatively easy to put in place and to automatize
✅ You don't need to install anything.
❌ A bit time-consuming to set up the first time

Now we can access our website on a real mobile device. But what if we need HTTPS ?

Using LocalTunnel to access your website in the local environment from your mobile with HTTPS

What is LocalTunnel?

LocalTunnel is a service that allows you to expose an endpoint on both a public HTTP and a public HTTPS address.

If you don't know localTunnel, maybe you are more familiar with ngrok its best-known counterpart. However, the main advantage of localTunnel over ngrok is that you don't have a limit on the number of endpoints you can create whereas you have to pay as soon as the second one for ngrok.

How to set it up?

You have two options to use LocalTunnel on your project

Start the tunnels using the command line using: lt --port <port>

Running localTunnel using the command line is pretty straightforward
Using the npm package directly in a node project

const localtunnel = require("localtunnel");
const myTunnel = await localtunnel({ port: 3000 });
console.log(myTunnel.url);

You will still have to update your URLs in your configuration file (same principles as for the local IP address). I personally prefer the second solution because the address returned by LocalTunnel is random and the task of updating the URL for your backend/frontend in your configuration files can quickly become annoying. I developed a quick example of an independent node project that sets up your configuration files when launching the app.

Concerning logs, they are still available using the same principles as for the local IP address.

The ultimate solution to local development on mobile?

Pros:

You are making your website accessible to anyone on the internet and not only on your local network anymore. You want your client/another developer/your manager to see the new feature before deploying it, this is now possible!
LocalTunnel provides an HTTPS address so you won't have any issue related to HTTP anymore.

Cons

Complex to set up. Each time you launch localTunnel, the addresses provided are different which makes it a bit tricky to set up with a script even though it remains possible.
You can only test on the devices/browser versions you have access to.
Even if there is little chance of anything happening you are still making your website public. Make sure you don't display any critical information.

Solution evaluation

📱 Close to end-user experience: Very Good

✅ Directly on your mobile device
✅ In HTTPS
❌ Only the mobile you own / the browser version you have access to

📋 Access to log: Very good - No Logs.

✅ The same quality of logs as when you are developing for computer users...
❌ ...Except for the IOS + Windows/Linux duo.

🔧 Easy to set up: So-so.

✅ Can be automated
❌ More time-consuming than the local-network solution

Now you can access to your website on a mobile device in HTTPS. But you may wonder what to do when you don't have the device / the browser version you want to check

Emulators/simulators

You may be tempted to use an emulator / a simulator to access your local website. Indeed, using Android Studios or Xcode Ios Simulator to access your website during local development is possible but here are three reasons why this is probably a bad idea:

Emulators are heavy. Your computer will soon be over-encumbered with mobile images
Emulators are slow. You have to imitate the behavior of smartphone components on your computer and it requires a lot of calculation power making your application slow/laggy in the emulator.
Emulators are not designed for web development. Accessing quality logs can be really hard.

If you want to test a specific mobile or a specific browser version, your best bet is probably to use BrowserStack.

Our decision tree is complete !

Takeaway

Two last things that can help you find the best solution for your project :

Don't go for the most technical solution if all you need is to test basic responsiveness. The chrome device tool is already very complete.
If you want to test complex behavior, you will probably know it beforehand. So set up your project for mobile testing at the beginning. You will lose far less time on the way.

How To Make Tree Shakeable Libraries

Tue, 13 Apr 2021 00:00:00 GMT

At Theodo, our aim is to build reliable and fast applications for our customers. Some of our projects include improving the performance of already existing applications. During one of these missions, we managed to reduce the bundle size of all our pages by a whopping 500KB Gzipped by tree shaking our internal libraries.

While doing so, we realized tree shaking is not something we can simply turn on and off. There are many factors that impact the tree shaking quality of libraries.

This article's aim is to provide an exhaustive guide on making an optimized tree shakeable library. If I had to summarize the procedure:

Check whether the library is tree shakeable by testing it against a known application in a controlled environment
Use ES6 modules so bundlers can detect unused export statements
Use the side effects optimization and make your library side effects free
Split the library logic in multiple small modules and preserve the library's module tree
Do not lose the module tree or the ES modules characteristics when transpiling your library
Use the latest version of a tree shaking capable bundler

What is tree shaking and why is it important?

From the MDN docs:

Tree shaking is a term commonly used within a JavaScript context to describe the removal of dead code.

It relies on the import and export statements in ES2015 to detect if code modules are exported and imported for use between JavaScript files.

Tree shaking is a way to achieve dead code elimination by detecting which exports are unused in our application. It is performed by application bundlers such as Webpack or Rollup but was initially implemented by Rollup.

So why call it tree shaking? One can visualize the application's exports and imports in the form of a tree. The healthy leaves and branches represent used imports of the application while dead leaves symbolize unused code separated from the rest of the tree. Shaking the tree would eliminate all our unused code.

Why is this important? Tree shaking can have a huge impact on your browser applications. The more code gets bundled in the app, the more time the browser will spend downloading, decompressing, parsing, and executing it. Removing unused code is thus of the utmost importance to make the fastest applications possible.

There are many articles and resources out there explaining tree shaking and dead code elimination. Here we will be focusing on libraries that are consumed by applications. A library is considered to be tree shakeable if a consumer application can successfully eliminate the unused parts of the library.

But before we try to make a library tree shakeable, let's first see how we can distinguish one.

Realize a library is not tree shakeable in a controlled environment

This may sound obvious at first glance. But I noticed a lot of developers assume their library is tree shakeable because it uses ES6 modules (more on that later) or because they have a tree shaking friendly configuration. Unfortunately, this does not automatically imply that your library is in fact tree shakeable!

This brings us to the relevant question: how can one efficiently check that a library is tree shakeable?

To do so, we need to understand two things:

It is the app's bundler that will ultimately tree shake the library's code, not the library's (if it even has a bundler at all!). After all, only the app knows which parts of our library are used.
The library's job is to make sure it can be tree shaken by the final bundler

To check whether our library is tree shakeable we will be testing it against a reference application in a controlled environment:

Create a simple application (reference app) with a bundler you know how to configure and that supports tree shaking (eg Webpack or Rollup)
Set the library you want to test as a dependency of the created application
Import only one element of the library and check the output of the application's bundler
Check that the bundled output only contains the imported element and its dependencies

This strategy will make the test independent of our existing applications. It makes it easier but also allows us to play with the library without breaking anything. It also makes sure the issue does not come from the application bundler configuration.

We will be doing this for our library called user-library that we will test against a user-app application bundled with Webpack. You may use whatever bundler you feel more comfortable with.

The user-library's code looks like this:

export const getUserName = () => "John Doe";

export const getUserPhoneNumber = () => "***********";

It just exports 2 functions in an index file that we can use via an NPM package.

Let's make our simple user-app:

package.json

{
  "name": "user-app",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "build": "webpack"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "webpack": "^5.18.0",
    "webpack-cli": "^4.3.1"
  },
  "dependencies": {
    "user-library": "1.0.0"
  }
}

Notice we are using user-library as a dependency.

webpack.config.js

const path = require("path");

module.exports = {
  entry: "./src/index.js",
  output: {
    filename: "main.js",
    path: path.resolve(__dirname, "dist"),
  },
  mode: "development",
  optimization: {
    usedExports: true,
    innerGraph: true,
    sideEffects: true,
  },
  devtool: false,
};

To understand the Webpack configuration of our reference application, we need to understand how Webpack does tree shaking. Tree shaking is performed using the following steps:

Identify the application entry file (determined in the Webpack configuration)
Create an application module tree by looping through all the dependencies and sub dependencies imported by the entry file
Identify for each module in the tree which export statements are not imported by the other modules
Eliminate the unused exports and their related code using minification tools like UglifyJS or Terser

This process in done only in production mode.

The issue with production mode is minification. It becomes very hard to see whether our tree shaking worked because we cannot actually see our named functions in our bundled code.

To get around this, we run Webpack in development mode while still identifying which code is unused and would be removed in production with the optimization property set to:

  optimization: {
    usedExports: true,
    sideEffects: true,
    innerGraph: true,
  }

The usedExports property allows Webpack to identify which module exports are not used by other modules. The other two will be discussed later in the article. For now, let's just say they improve the tree shaking quality of our application.

Our user-app entry file: src/index.js

import { getUserName } from "user-library";

console.log(getUserName());

Once bundled, we now analyze the output:

/***/ "./node_modules/user-library/dist/index.js":
/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((__unused_webpack_module, exports) => {

var __webpack_unused_export__;

__webpack_unused_export__ = ({ value: true });

const getUserName = () => 'John Doe';

const getUserPhoneNumber = () => '***********';

exports.getUserName = getUserName;
__webpack_unused_export__ = getUserPhoneNumber;
/***/ })

Webpack regroups all our code in a single file. Looking at the getUserPhoneNumber export, we notice that Webpack has marked it as unused. It will be removed in production mode while getUserName is exported as it is used by our index.js entry file.

The library is tree shaken! You may repeat this step but for multiple imports and look at the output code. The objective is to make sure unused code in the library is marked as unused by Webpack.

Everything looks fine for our very simple user-library. Let's make it a bit more complicated and as we do so, we will look at some tree shaking requirements and optimizations.

Use ES6 modules so bundlers can detect unused exports

Ok, this requirement is classic and well documented but is in my opinion a bit misleading. I oftentimes hear developers say we need to use ES6 modules so that our library can be tree shaken. While this statement is completely true, there is this misconception that using ES6 modules is enough to just make tree shaking work. Well, were it that simple, you definitely would not have gotten this far in the article!

However, using ES6 modules still is a requirement for tree shaking.

There are a lot of formats in which JavaScript can be bundled: ESM, CJS, UMD, IIFE and so on.

To make things simple, we will consider only two: Ecma Script Modules (ESM or ES6 modules) and CommonJS modules (CJS) as they are the most widely used for application libraries. Most libraries will use CJS modules because it allows them to be run in a node application (though Node now supports ESM). ES modules appeared way later than CJS in 2015 with ECMAScript 2015 (also known as ES6) and is considered to be the standardized module system for JavaScript.

CJS syntax example

const { userAccount } = require("./userAccount");

const getUserAccount = () => {
  return userAccount;
};

module.exports = { getUserAccount };

ESM syntax example

import { userAccount } from "./userAccount";

export const getUserAccount = () => {
  return userAccount;
};

The big difference between both is that ESM imports are static whereas CJS imports are dynamic meaning we could do the following with CJS but not with ESM:

if (someCondition) {
  const { userAccount } = require("./userAccount");
}

While this seems to be more flexible, it also means bundlers cannot make a valid application tree at compile or bundle time. The someCondition variable will be known only at runtime forcing the bundler to import userAccount in any case at compile time. This leads bundlers to just include all the CJS style imports directly in the bundle without being able to check whether the imports are actually used.

Let's modify our user-library to show this. And to make it a little bit more realistic, it will now have two files:

src/userAccount.js

const userAccount = {
  name: "user account",
};

module.exports = { userAccount };

src/index.js

const { userAccount } = require("./userAccount");

const getUserName = () => "John Doe";

const getUserPhoneNumber = () => "***********";

const getUserAccount = () => userAccount;

module.exports = {
  getUserName,
  getUserPhoneNumber,
  getUserAccount,
};

We keep the same entry file in our user-app so we do not use the getUserAccount function nor its dependency.

/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((module, __unused_webpack_exports, __webpack_require__) => {

const { userAccount } = __webpack_require__(/*! ./userAccount */ "./node_modules/user-library/dist/userAccount.js")

const getUserName = () => 'John Doe'

const getUserPhoneNumber = () => '***********'

const getUserAccount = () => userAccount

module.exports = {
	getUserName,
	getUserPhoneNumber,
	getUserAccount
}
/***/ }),

/***/ "./node_modules/user-library/dist/userAccount.js":
/*!*******************************************************!*\
  !*** ./node_modules/user-library/dist/userAccount.js ***!
  \*******************************************************/
/***/ ((module) => {

const userAccount = {
	name: 'user account'
}

module.exports = { userAccount }
/***/ })

All three exports still appear and are not marked by Webpack as unused. This is also the case for our userAccount file which will be included in the bundle.

Now let's look at the same example but with ESM by just replacing the require and exports syntax with their ESM counterparts.

/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {
/* harmony export */ __webpack_require__.d(__webpack_exports__, {
/* harmony export */   "getUserName": () => /* binding */ getUserName
/* harmony export */ });
/* unused harmony exports getUserAccount, getUserPhoneNumber */
/* harmony import */ var _userAccount_js__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(/*! ./userAccount.js */ "./node_modules/user-library/dist/userAccount.js");

const getUserName = () => 'John Doe';

const getUserPhoneNumber = () => '***********';

const getUserAccount = () => userAccount;

/***/ }),
/***/ "./node_modules/user-library/dist/userAccount.js":
/*!*******************************************************!*\
  !*** ./node_modules/user-library/dist/userAccount.js ***!
  \*******************************************************/
/***/ ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {

/* unused harmony export userAccount */
const userAccount = {
	name: 'user account'
};
/***/ })

Notice that getUserAccount and getUserPhoneNumber are marked as unused. But so does the userAccount export in the other file. Thanks to the innerGraph optimization, Webpack is able to link the userAccount import in the index file to the getUserAccount export. This allows Webpack to work recursively from the entry file and go through all of its dependencies to know which exports are unused in every module. Since Webpack knows that getUserAccount is unused, it can go and check its dependencies in the userAccount file and do the same work there etc.

ES modules allow us to look for exported code that is used or unused in our application explaining why this module system is so important for tree shaking. It also explains why one should use dependencies that export a ES module compatible build such as lodash-es, the ESM equivalent of the popular lodash library.

This being said, using only ES modules for tree shaking is still a suboptimal approach. In our example, we noticed Webpack worked recursively in each file to see whether exported code was used or unused. In this case, Webpack could have just totally ignored the userAccount file because the only import coming from that file is unused! This leads us to the notion of side effects discussed in the next part of the article.

To sum up this first part:

ESM is a requirement for tree shaking but is not enough to make it optimal
Make sure to always provide an ESM build of your library! If your consumers require both ESM and CJS builds, provide them both through the package.json with the main and module fields.
Make sure to use ESM dependencies when possible as they won't be tree shaked otherwise.

Use the side effects optimization to make your library side effects free

According to the Webpack docs, tree shaking can be split in two optimizations:

usedExports: Determine which exports of a module are used and unused
sideEffects: skip over modules that do not have any used exports and that are side effects free

To illustrate side effects, let's take the example that we used before:

import { userAccount } from "./userAccount";

function getUserAccount() {
  return userAccount;
}

If getUserAccount is unused, can the bundler assume that the userAccount module can be removed as well? The answer is no! userAccount could do all kinds of stuff affecting other parts of the app. It could inject some variables inside a globally accessible value, like the DOM for instance. It could also be a css module injecting style inside the document. But I think the best example is polyfills as we usually import them like:

import "myPolyfill";

Now this module definitely has side effects as it affects the code of the whole app as soon as it is imported. Bundlers will see this module as a possible candidate for removal as we are not using any of its exports. But removing this would break our app.

Bundlers like Webpack or Rollup will therefore see every module of our library as filled with side effects by default.

In our case, we know that our library is side effect free! We can therefore help our bundler by informing it of this. To do so, most bundlers can read the sideEffects property inside the package.json file. It is by default set to true when unspecified (there are side effects in every module of the package). You can set it to false (no side effects anywhere) or you may specify an array of files that have side effects.

We add this option to the package.json file of our library:

{
  "name": "user-library",
  "version": "1.0.0",
  "description": "",
  "sideEffects": false,
  "main": "dist/index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}

We then rerun our Webpack build:

/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ (__unused_webpack_module, __webpack_exports__, __webpack_require__) => {
  /* harmony export */ __webpack_require__.d(__webpack_exports__, {
    /* harmony export */ getUserName: () => /* binding */ getUserName,
    /* harmony export */
  });
  /* unused harmony exports getUserAccount, getUserPhoneNumber */

  const getUserName = () => "John Doe";

  const getUserPhoneNumber = () => "***********";

  const getUserAccount = () => userAccount;
  /***/
};

We see that the userAccount file has been removed from the bundle. We still see getUserAccount that references userAccount but this function has been marked by Webpack as dead code and it will be removed during minification.

The sideEffects flag is especially important for libraries that export their API through an index file that itself exports functions or variables from internal files. Without the side effects optimization, our bundlers would have to parse all the files where our exported variables are defined.

As noted by Webpack: "sideEffects is much more effective since it allows to skip whole modules/files and the complete subtree."

To better understand the difference between how both optimizations intervene:

sideEffects allows us to completely skip an imported module if none of its imported contents are used.
usedExports allows us to completely remove exports that are never imported by any module.

Ok but how is the result of skipping files different from just saying the exports of those files are unused?

Most of the time, tree shaking a library with and without the side effects optimization will give the same result. The same amount of code will be included in the final bundle. However, this is not true in some situations when analyzing the code for unused exports becomes too complex. The next part covers two of these situations where only the combination of small modules with the sideEffects optimization provides the best result.

To sum up this part:

Tree shaking consists of two parts: the used exports optimization and the side effects optimization
The side effects optimization is way more effective than just detecting unused export statements in every module
Make your library side effect free
Make sure to tell that your library is side effects free through the sideEffects property in the package.json file.

Preserve the library's module tree and split your code in small modules to fully benefit from the `sideEffects` optimization

You may have noticed that the example user-library we are using in this article is not bundled. The library just exposes a few JS files that I manually added.

Oftentimes, libraries will be bundled for multiple reasons:

Manage custom import paths
The library uses specific languages like SASS or TS that need to be transformed to CSS or JS for instance
The library needs to be available in multiple formats (ESM, CJS, IIFE ...)

Popular bundlers like Webpack, Rollup, Parcel or ESBuild are made to provide a bundle that can be served to a browser. They will thus have a tendency to create a single file regrouping all of your code so that only a single JS file needs to be sent through the network.

From a tree shaking perspective, this creates one problem: The side effects optimization is non existent as no modules can be skipped.

We are going to showcase two situations where splitted modules combined with the sideEffect optimization are essential for tree shaking.

A library module imports a CJS formatted dependency

To demonstrate the issue, we are going to bundle our library using Rollup. And we are going to have one of our library modules import a CJS formatted dependency: Lodash

rollup.config.js

export default {
  input: "src/index.js",
  output: {
    file: "dist/index.js",
    format: "esm",
  },
};

userAccount.js

import { isNil } from "lodash";

export const checkExistance = (variable) => !isNil(variable);

export const userAccount = {
  name: "user account",
};

Notice we are now exporting checkExistance and we import it in our library index file.

Here is the output in dist/index.js

import { isNil } from "lodash";

const checkExistance = (variable) => !isNil(variable);

const userAccount = {
  name: "user account",
};

const getUserAccount = () => {
  return userAccount;
};

const getUserPhoneNumber = () => "***********";

const getUserName = () => "John Doe";

export { checkExistance, getUserName, getUserPhoneNumber, getUserAccount };

Everything is bundled within a single file. Notice also that Lodash is imported at the top. We are still importing the same functions in our application meaning checkExistance is not used. However, after running Webpack, we see that the whole Lodash library is imported even though checkExistance is marked as unused:

/***/ "./node_modules/user-library/dist/index.js":
/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {

"use strict";
/* harmony export */ __webpack_require__.d(__webpack_exports__, {
/* harmony export */   "getUserName": () => (/* binding */ getUserName)
/* harmony export */ });
/* unused harmony exports checkExistance, userAccount, getUserPhoneNumber, getUserAccount */
/* harmony import */ var lodash__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(/*! lodash */ "./node_modules/user-library/node_modules/lodash/lodash.js");
/* harmony import */ var lodash__WEBPACK_IMPORTED_MODULE_0___default = /*#__PURE__*/__webpack_require__.n(lodash__WEBPACK_IMPORTED_MODULE_0__);

const checkExistance = (variable) => !isNil(variable);

const userAccount = {
  name: "user account",
};

const getUserPhoneNumber = {
	number: '***********'
};

const getUserAccount = () => {
	return userAccount
};

const getUserName = () => 'John Doe';

/***/ }),

/***/ "./node_modules/user-library/node_modules/lodash/lodash.js":
/*!*****************************************************************!*\
  !*** ./node_modules/user-library/node_modules/lodash/lodash.js ***!
  \*****************************************************************/
/***/ (function(module, exports, __webpack_require__) {

/* module decorator */ module = __webpack_require__.nmd(module);
var __WEBPACK_AMD_DEFINE_RESULT__;/**
 * @license
 * Lodash <https://lodash.com/>
 * Copyright OpenJS Foundation and other contributors <https://openjsf.org/>
 * Released under MIT license <https://lodash.com/license>
 * Based on Underscore.js 1.8.3 <http://underscorejs.org/LICENSE>
 * Copyright Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors
 */
// ...

Webpack is not able to tree shake Lodash because of its CJS format. This is a shame as we explicitly organized our library so Lodash would be imported only in the userAccount module that is unused in our application. If the module structure was preserved, Webpack would have detected that no userAccount exports were used and would simply have skipped the module and thus its Lodash import thanks to the sideEffects optimization.

In rollup, we can use the preserveModules option to preserve the module structure of our library.. Equivalents exist for other bundlers.

export default {
  input: "src/index.js",
  output: {
    dir: "dist",
    format: "esm",
    preserveModules: true,
  },
};

Rollup now keeps the original file structure. We can now run Webpack again:

/***/ "./node_modules/user-library/dist/index.js":
/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {

/* harmony export */ __webpack_require__.d(__webpack_exports__, {
/* harmony export */   "getUserName": () => (/* binding */ getUserName)
/* harmony export */ });
/* unused harmony export getUserAccount */

const getUserAccount = () => {
	return userAccount
};

const getUserName = () => 'John Doe';

/***/ })

Lodash is now skipped along with the userAccount module.

Code splitting

Preserving splitted module structure alongside the sideEffects optimization also benefits Webpack code splitting, a key performance optimization for big applications. Code splitting is widely used in web applications with multiple pages. Frameworks like Nuxt or Next both use page by page code splitting.

To illustrate the benefit, we will look at what happens when the library is bundled in a single file.

user-library/src/userAccount.js

export const userAccount = {
  name: "user account",
};

user-library/src/userPhoneNumber.js

export const userPhoneNumber = {
  number: "***********",
};

user-library/src/index.js

import { userAccount } from "./userAccount";
import { userPhoneNumber } from "./userPhoneNumber";

const getUserName = () => "John Doe";

export { userAccount, getUserName, userPhoneNumber };

In order to code split our user application, we will use the Webpack import syntax.

user-app/src/userService1.js

import { userAccount } from "user-library";

export const logUserAccount = () => {
  console.log(userAccount);
};

user-app/src/userService2.js

import { userPhoneNumber } from "user-library";

export const logUserPhoneNumber = () => {
  console.log(userPhoneNumber);
};

user-app/src/index.js

const main = async () => {
  const { logUserPhoneNumber } = await import("./userService2");
  const { logUserAccount } = await import("./userService1");

  logUserAccount();
  logUserPhoneNumber();
};

main();

The app bundle now has 3 files: main.js, src_userService1_js.main.js and src_userService2_js.main.js. Taking a closer look at src_userService2_js.main.js, we can see that the entire user-library bundle is added:

(self["webpackChunkuser_app"] = self["webpackChunkuser_app"] || []).push([
  ["src_userService1_js"],
  {
    /***/ "./node_modules/user-library/dist/index.js":
      /*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
      /***/ (
        __unused_webpack_module,
        __webpack_exports__,
        __webpack_require__
      ) => {
        "use strict";
        /* harmony export */ __webpack_require__.d(__webpack_exports__, {
          /* harmony export */ userAccount: () => /* binding */ userAccount,
          /* harmony export */ userPhoneNumber: () =>
            /* binding */ userPhoneNumber,
          /* harmony export */
        });
        /* unused harmony export getUserName */
        const userAccount = {
          name: "user account",
        };

        const userPhoneNumber = {
          number: "***********",
        };

        const getUserName = () => "John Doe";

        /***/
      },

    /***/ "./src/userService1.js":
      /*!*****************************!*\
  !*** ./src/userService1.js ***!
  \*****************************/
      /***/ (
        __unused_webpack_module,
        __webpack_exports__,
        __webpack_require__
      ) => {
        "use strict";
        __webpack_require__.r(__webpack_exports__);
        /* harmony export */ __webpack_require__.d(__webpack_exports__, {
          /* harmony export */ logUserAccount: () =>
            /* binding */ logUserAccount,
          /* harmony export */
        });
        /* harmony import */ var user_library__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(
          /*! user-library */ "./node_modules/user-library/dist/index.js"
        );

        const logUserAccount = () => {
          console.log(user_library__WEBPACK_IMPORTED_MODULE_0__.userAccount);
        };

        /***/
      },
  },
]);

userAccount is not marked as unused even though userService2 only uses userPhoneNumber... But why ?

We need to keep in mind that the usedExports optimization checks for used exports only within a module's scope. Only from there can Webpack remove unused code. From the perspective of our library module, both userAccount and userPhoneNumber are actually used. In this case, Webpack is not able to make a difference between the imports of userService1 and userService2 as seen on the following graph (both userAccount and userPhoneNumber are in green):

This means that Webpack is not able to tree shake the exports of each chunk independently when only relying on the usedExports optimization.

We now preserve our modules when bundling our library to allow for the sideEffects optimization:

Webpack still outputs the same 3 files but this time, src_userService2_js.main.js only contains the code coming from userPhoneNumber:

(self["webpackChunkuser_app"] = self["webpackChunkuser_app"] || []).push([
  ["src_userService2_js"],
  {
    /***/ "./node_modules/user-library/dist/userPhoneNumber.js":
      /*!***********************************************************!*\
  !*** ./node_modules/user-library/dist/userPhoneNumber.js ***!
  \***********************************************************/
      /***/ (
        __unused_webpack_module,
        __webpack_exports__,
        __webpack_require__
      ) => {
        "use strict";
        /* harmony export */ __webpack_require__.d(__webpack_exports__, {
          /* harmony export */ userPhoneNumber: () =>
            /* binding */ userPhoneNumber,
          /* harmony export */
        });
        const userPhoneNumber = {
          number: "***********",
        };

        /***/
      },

    /***/ "./src/userService2.js":
      /*!*****************************!*\
  !*** ./src/userService2.js ***!
  \*****************************/
      /***/ (
        __unused_webpack_module,
        __webpack_exports__,
        __webpack_require__
      ) => {
        "use strict";
        __webpack_require__.r(__webpack_exports__);
        /* harmony export */ __webpack_require__.d(__webpack_exports__, {
          /* harmony export */ logUserPhoneNumber: () =>
            /* binding */ logUserPhoneNumber,
          /* harmony export */
        });
        /* harmony import */ var user_library__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(
          /*! user-library */ "./node_modules/user-library/dist/userPhoneNumber.js"
        );

        const logUserPhoneNumber = () => {
          console.log(
            user_library__WEBPACK_IMPORTED_MODULE_0__.userPhoneNumber
          );
        };

        /***/
      },
  },
]);

src_userService1_js.main.js behaves the same way as it includes only the userAccount module from our library.

Looking at the graph, we still see that userAccount and userPhoneNumber are still considered as used exports as they are used at least once in our application. However, this time the sideEffects optimization is able to skip the userAccount module because it is never imported by userService2. The same thing happens for userPhoneNumber and userService1.

We can now understand that preserving the original module structure of the library is important. However, preserving this is useless if the original structure only has one module such as an index.js file with all the code inside. In order to make an optimal tree shakeable library, its code should be divided in multiple small modules that each handle one piece of the logic.

To use the tree analogy, one should see each leaf of the tree as a module. Smaller and weaker leafs will fall better when the tree is shaked! If the tree has fewer and bigger leafs, shaking it will not give the same result.

To sum up this part:

We should preserve the module structure of the library in order to fully benefit from the sideEffects optimization.
Libraries should be split in multiple small modules, each module exporting only one piece of logic
Tree shaking libraries in applications that use code splitting will only work with the sideEffects optimization.

Do not lose the module tree or the ES modules characteristics when transpiling your library

Bundlers are not the only tools that can harm the tree shaking of your library. Transpilers are also known to have an undesirable effect on tree shaking by removing ES modules and by not preserving the module tree.

One of the objectives of transpilers is to make your code compatible for browsers that do not necessarily support ES modules. We should remember though that our libraries are not meant to be served to browsers directly but should instead be consumed by applications. So transpiling our libraries to target specific browsers should be forbidden for two reasons:

When making a library, one does not know which browser should be targeted, only the application knows that.
Transpiling libraries can make them non tree shakeable

If the library needs to be transpiled for some reason, one needs to make sure the transpilation does not remove the ES module syntax or the original module tree for the same reasons explained in the last part of the article.

There are two tools that I know of that transpiled my libraries removing their tree shakeable characteristic.

Babel

Babel uses Babel preset-env to make your code compatible with one's target browsers. By default, this plugin will remove ES modules from the library. To make sure this does not happen, set the modules option to false:

module.exports = {
  env: {
    esm: {
      presets: [
        [
          "@babel/preset-env",
          {
            modules: false,
          },
        ],
      ],
    },
  },
};

Typescript

When compiling your code, typescript will transform your modules depending on the target and module options you set in the tsconfig.json file.

To make sure this does not happen, set the target and module options to at least ES2015 or ES6.

To sum up:

Make sure your transpilers/compilers do not remove the ES module syntax from your library bundle.
To check whether this happens, look at the library's output bundle and check for ESM import syntax.

Use the latest version of a tree shaking capable bundler

Tree shaking in javascript was popularized by Rollup. Webpack supports this since version 2 and bundlers keep getting better and better at optimizing tree shaking.

Remember when we talked about the innerGraph optimization that allows Webpack to link module exports to the module's imports? This optimization was introduced in Webpack 5. We have been using Webpack 5 in this article but it is important to note that this optimization is a game changer as it allows Webpack to recursively look for unused exports!

To show what it actually does, we can consider our index.js file in our user-library:

import { userAccount } from "./userAccount";

const getUserAccount = () => {
  return userAccount;
};

const getUserName = () => "John Doe";

export { getUserName, getUserAccount };

Our user-app only uses getUserName.

import { getUserName } from "user-library";

console.log(getUserName());

We can now compare the outputs with and without the innerGraph optimization. We are still using both the usedExports and sideEffects optimizations:

Without the innerGraph optimization (eg with Webpack 4):

/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/*! exports provided: userAccount, userPhoneNumber, getUserName, getUserAccount */
/*! exports used: getUserName */
/***/ (function(module, __webpack_exports__, __webpack_require__) {

"use strict";
/* harmony export (binding) */ __webpack_require__.d(__webpack_exports__, "a", function() { return getUserName; });
/* unused harmony export getUserAccount */
/* harmony import */ var _userAccount_js__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(/*! ./userAccount.js */ "./node_modules/user-library/dist/userAccount.js");

const getUserAccount = () => {
	return _userAccount_js__WEBPACK_IMPORTED_MODULE_0__[/* userAccount */ "a"]
};

const getUserName = () => 'John Doe';

/***/ }),

/***/ "./node_modules/user-library/dist/userAccount.js":
/*!*******************************************************!*\
  !*** ./node_modules/user-library/dist/userAccount.js ***!
  \*******************************************************/
/*! exports provided: userAccount */
/*! exports used: userAccount */
/***/ (function(module, __webpack_exports__, __webpack_require__) {

"use strict";
/* harmony export (binding) */ __webpack_require__.d(__webpack_exports__, "a", function() { return userAccount; });
const userAccount = {
	name: 'user account'
};

/***/ }),

With the innerGraph optimization (eg with Webpack 5):

/***/ "./node_modules/user-library/dist/index.js":
/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {

/* harmony export */ __webpack_require__.d(__webpack_exports__, {
/* harmony export */   "getUserName": () => (/* binding */ getUserName)
/* harmony export */ });
/* unused harmony export getUserAccount */

const getUserAccount = () => {
	return userAccount
};

const getUserName = () => 'John Doe';

/***/ })

While Webpack 5 is able to completely eliminate the userAccount module, this is not the case for Webpack 4 even though getUserAccount is marked as unused. This is because the innerGraph algorithm allows webpack 5 to link unused elements of our module with its imports. In our case, the userAccount module is used only by the getUserAccount function and can therefore be skipped.

This optimization does not work using Webpack 4. Developers should therefore be careful and limit the number of exports in a single file when using this version of Webpack. I a file contains multiple exports, Webpack will include all the file imports even though they may not be necessary for the desired export.

In general, we should make sure our bundlers are always up to date to benefit from their latest tree shaking optimizations.

Conclusion

Tree shaking a library is not something one just turns on by adding a specific line in a configuration file. Its quality depends on multiple factors and this article presents only a few of them. In the end though, whatever the issue might be, there are two important things we did in this article that can help anyone tree shake their libraries:

In order to see how well our library is tree shaked, we test it in a controlled environment using a bundler we know how to use.
We detect issues with our library setup not by looking at its configuration files but by inspecting its bundled output. This is what we have been doing all along in this article with our user-library and user-app examples.

I really hope this article helps you in your evergoing quest to make the best and most optimized libraries possible!

An introduction to DynamoDB data modeling

Mon, 12 Apr 2021 00:00:00 GMT

I discovered DynamoDB a few months ago, in the context of a very high traffic web application built using the Serverless Framework. DynamoDB - as well as Single-Table design - was fairly new to me, and I quickly noticed that there are a few concepts to grasp before designing the first data models. This article aims to provide a beginner-friendly tutorial for developers who are considering starting a project with DynamoDB, or who are just curious about this technology.

DynamoDB is a fully managed NoSQL database released by AWS in 2012. As the other main NoSQL solutions, such as MongoDB (2009) or Apache Cassandra (2008), it was designed at a time when storage prices were low compared to those of processing power. In the seventies, when relational databases were first introduced, this relationship was inverted. In particular, high storage prices gave incentives to limit storage needs by avoiding data duplication. You would design separate entities and put each one of them in a separate table, which drastically simplifies the data modeling exercise. You could then simply write the entity-relationship diagram (ERD), translate it into tables and retrieve the information of your choice by joining tables together.

With a NoSQL database such as DynamoDB, the join functionality does not exist, you're invited to put all your entities in the same table. This is the concept of Single-Table Design. By removing joins, you write more efficient queries, which reduces the needs of computing power. However, this comes at a cost: the data modeling exercise is slightly more complicated. The entity diagram does not give you the model anymore. The ways your application will interact with data - the access patterns - are now a key building block of your modelisation.

In this tutorial, I will take the example of a sample book review website. On the application, users can write a review for a book of their choice. They can also read the reviews written by others on any given book. Finally, users can read all the reviews written by a given user. As a bonus, we will talk about sorting these reviews in ante-chronological order.

Start from an entity-relationship diagram

The starting point of the exercise is the ERD. We won't need any joins here, which are the reflection of relationships between entities, but having these relationships in mind is necessary to know how you'll make it work without them.

In our case, the diagram is quite simple. There are 3 entities: books, users and reviews. There is a one-to-many relationship between users and reviews (a user can write several reviews), and another one-to-many relationship between reviews and books (a book can be reviewed several times). In this case, reviews implicitly represent the many-to-many relationship between users and books.

Entity-relationship diagram for users, books, and reviews.

As described on the diagram, we use the ISBN (International Standard Book Number) as a unique identifier for books, and the username for users. A review consists of a date, a grade, and a text.

Write down the access patterns

Before any attempt to model the keys in your table, you should write down the access patterns of your application. The modelisation's purpose is to handle all of them. It is important to be thorough in this step, or you might be unable to retrieve some information from your table. For the purpose of this exercise, let's consider three of them.

Access Pattern
Get all the reviews for a given book
Get all the reviews for a given user
Update a review

On the book page, we need the reviews for this particular book. On a user profile, we will query for the reviews written by this user. Finally, we will see how we would update a review.

Model the structure of the primary key

Once you have defined all of your access patterns, you can start the modelisation. The primary key is a core concept in DynamoDB, and almost the single piece of structure imposed to your table. It is thus at the core of the modeling exercise.

DynamoDB allows two kinds of primary keys:

Simple primary keys, made of a single element called partition key (PK).
Composite primary keys, made of partition key (PK) and sort key (SK).

The partition key is particularly important, as it represents an identifier of the server on which the data actually is (also called a storage node). Two items that have distinct partition keys will thus be difficult to query together.

The primary key related rules are simple:

A table must have a primary key
All the items in the table must have a value for that primary key
Each primary key must be unique <> an item is uniquely identified by its primary key
Each request to READ item(s) must specify the partition key *
Each request to WRITE an item must specify the full primary key

* Here I intentionally avoid scans, which are 9 times out of 10 bad practice

With these concepts in mind, you can start modeling. For each entity, we have to think about the best partition key and sort key to fulfill all the access patterns. In other words, the goal of this step is to fill the table below:

Entity	PK	SK
Book	`?`	`?`
Review	`?`	`?`
User	`?`	`?`

A common practice is to start with the top-level entity that has the highest number of relationships. Here, books and users play a somehow symmetric role so let's start with modeling books.

The goal is to be able to query at the same time for a book and its associated reviews. We thus have to make sure that the book reviews have the same partition key as the book itself, in order for them to be stored on the same physical machine. Using this partition key, DynamoDB will identify the associated storage node in a O(1) time. In DynamoDB terminology, we say that we put the book and its reviews in the same item collection.

Because an item in DynamoDB must be uniquely identified by its primary key, the sort key will be the way we differentiate a book from its reviews. Let's have a look at the following primary key structure:

Entity	PK	SK
Book	`BOOK#<ISBN Number>`	`BOOK#<ISBN Number>`
Review	`BOOK#<ISBN Number>`	`USER#<Username>`

Prefixing the key by the entity name is a common pattern. It gives us more clarity and compensates for the fact that we put all entities in the same table. The ISBN of the book is here used as a unique id. It enforces the uniqueness of a book's primary key. Finally, the sort key USER#<Username> allows us to differentiate reviews. It also identifies the author of the review. Implicitly, it says that a given user can not write two reviews about the same book (otherwise it violates the primary key's uniqueness principle).

Visualize your model in NoSQL Workbench

During the modeling exercise, it is good practice to model out some example data. It especially helps to visualize how a query would respond. The NoSQL Workbench is a great tool for that. At this stage of the modelisation, here is how a simple example would look like:

Main index - Books and reviews visualisation in NoSQL Workbench.

A row in this table is an item in DynamoDB. Both items share a common partition key, they are in the same item collection and can thus be queried together. We choose generic names for the key attributes (PK and SK) because they do not store the same type of information depending of the kind of entity they belong to.

To solve the first access pattern, just make a request for the items with a PK of BOOK#<ISBN>, with no condition on the sort key (remember that for a READ operation, specifying the sort key is optional). Both items in the table match this condition, so they will be returned together, sorted alphabetically on the sort key. In JSON format, you get something like

[
  {
    "PK": {
      "S": "BOOK#<ISBN>"
    },
    "SK": {
      "S": "BOOK#<ISBN>"
    },
    "Author": {
      "S": "J.R.R. Tolkien"
    },
    "Title": {
      "S": "The Fellowship of the Ring"
    },
    "PublicationYear": {
      "N": "1954"
    }
  },
  {
    "PK": {
      "S": "BOOK#<ISBN>"
    },
    "SK": {
      "S": "USER#<username>"
    },
    "Grade": {
      "N": "10"
    },
    "ReviewText": {
      "S": "Must read."
    },
    "ReviewDate": {
      "S": "2001-08-05T02:46:00"
    }
  }
]

Define secondary indexes if needed

Let's move on to the second access pattern. We want to be able to query for a user and all their reviews. We therefore need the user to be in the same partition as their reviews. But these already share a common partition key with the book they relate to. We thus somehow need the reviews to be in two distinct item collections. This is the notion of global secondary index (GSI).

When creating a GSI, you specify two attributes in your table that will act as a primary key in the secondary index. In other words, this index will be a copy of the data of the main table, in which the primary key is different. Using this, we can have the book and reviews to share a common PK in the main index, while having the reviews and the associated author in the same partition in a secondary index.

In our case, we can then adopt symmetric approaches in the main and secondary index. In the main index, the primary key structure will be like below:

Entity	PK	SK
Book	`BOOK#<ISBN Number>`	`BOOK#<ISBN Number>`
Review	`BOOK#<ISBN Number>`	`USER#<Username>`
User	`USER#<Username>`	`USER#<Username>`

See that the primary key structure for books and users follow the exact same pattern. In NoSQL Workbench, adding the review author to our example would bring us to:

Main index - Users, books and reviews visualisation.

Notice that both the user and their review have a SK of USER#<Username>. Having the SK of the main index as partition key in the secondary one would therefore solve the second access pattern. More precisely, we can just flip the PK and SK of the main index in the GSI. The secondary index in NoSQL Workbench would then be:

Secondary index - Users, books and reviews visualisation.

To solve the second access pattern, we make a request on the secondary index for the items with a partition key (here the SK attribute) of USER#<Username>, with no condition on the sort key. Both the user and the review match this condition, so they will be returned together in an array, sorted alphabetically on the sort key. To retrieve the user first, we can set the scanIndexForward property to false, so that DynamoDB reads the items in descending order. Note that by default (with scanIndexForward property set to true), DynamoDB will send the items in the order they are sorted in the table, which is the ascending order. If you are unsure, just look at your items in the NoSQL Workbench: top to bottom is the ascending ordering used by DynamoDB.

At this stage of the modelisation, most of the work is done. Inserting a review - the third access pattern - is trivial. We use the UpdateItem action of DynamoDB (you can learn more about writing items here), by specifying BOOK#<ISBN> for the partition key and USER#<Username> for the sort key (remember that for WRITE operations you must provide both the partition key and the sort key), along with the other attributes characterizing the review - grade, review text and date. By doing this, you implicitly specify a value for the PK and SK in the secondary index, and DynamoDB will thus take care of updating that item in the GSI as well.

What about sorting?

Sorting the query results before they are sent back to your application can be crucial in some situations. Going back to the book review example, imagine that you want to paginate the reviews for a particular book. More precisely, what if you want to show the 1 to X most recent reviews on the first page, then the X+1 to 2X most recent ones on the second page etc... Without sorting, you would have to retrieve all reviews on each page, and then get rid of those that are not in the appropriate range for that page. This would be very inefficient, especially if your application has a lot of contributors.

Let's cover how we would sort the reviews in ante-chronological order. As we have seen so far, in DynamoDB, almost everything is achieved using the primary key. It is also true for sorting. If you want a special ordering of the items returned from a query, there is only one way to do so: it has to be done with the sort key. Implicitly, it means that you need to arrange your items so that they are sorted in advance. This again stresses the need for thorough data modeling upfront.

In our case, the sort key thus needs to cary some sort of time indicator. Both unix timestamps (e.g. 744569696) or ISO-8601 date format (e.g. 2021-03-05T18:54:56+00:00) would work. The only requirement is that it must be sortable. Let's go for ISO- 8601 timestamps as they are human-readable. We can then prefix the sort key for reviews like so:

Main index - Timestamped version - Users, books and reviews visualisation.

For the review entity, the partition key and sort key in the secondary index are no longer an inversion of the primary key in the main table. That's why we have to add specific attributes to our table. We name them generically GSIPK and GSISK respectively as they do not store the same information per entity. They will act as partition and sort key in the secondary index.

To get a book and the associated reviews sorted, the query is the same as before. Query all the items with a partition key of BOOK#<Username> with no restrictions on the sort key. With the scanIndexForward property set to false, you would retrieve the book and then the review. As DynamoDB orders the items in ascending orders in your table, a more recent review would come after the first one:

Main index - Timestamped version - Sorted reviews.

With the scanIndexForward property set to false, you retrieve the items from bottom to top. This gives you the book and then the reviews from most recent to oldest. Finally, use the limit option (like in SQL) set to 16 for example and you restrict yourself to the 15 most recent reviews.

In the secondary index, the data is structured symmetrically. In this index, the item attributes GSI1PK and GSISK are the partition and sort key, respectively.

Secondary index - Timestamped version - Sorted reviews.

A query for items with a partition key (here the GSIPK attribute) of USER#<Username> with no restriction on the sort key gives us the user and their reviews. With scanIndexForward property set to false, the user will come first, and then their reviews in ante-chronological order. The response, in JSON format would be:

[
  {
    "PK": {
      "S": "USER#<Username>"
    },
    "SK": {
      "S": "USER#<Username>"
    },
    "MembershipDate": {
      "S": "2000-09-29T04:31:00"
    },
    "GSI1PK": {
      "S": "USER#<Username>"
    },
    "GSI1SK": {
      "S": "USER#<Username>"
    }
  },
  {
    "PK": {
      "S": "BOOK#<ISBN>"
    },
    "SK": {
      "S": "2002-08-05T02:46:00#USER#<Username>"
    },
    "Grade": {
      "N": "5"
    },
    "ReviewText": {
      "S": "Fantastic"
    },
    "ReviewDate": {
      "S": "2002-08-05T02:46:00"
    },
    "GSI1PK": {
      "S": "USER#<Username>"
    },
    "GSI1SK": {
      "S": "2002-08-05T02:46:00#BOOK#<ISBN>"
    }
  },
  {
    "PK": {
      "S": "BOOK#<ISBN>"
    },
    "SK": {
      "S": "2001-08-05T02:46:00#USER#<Username>"
    },
    "Grade": {
      "N": "5"
    },
    "ReviewText": {
      "S": "Must read."
    },
    "ReviewDate": {
      "S": "2001-08-05T02:46:00"
    },
    "GSI1PK": {
      "S": "USER#<Username>"
    },
    "GSI1SK": {
      "S": "2001-08-05T02:46:00#BOOK#<ISBN>"
    }
  },
]

Wrapping it all up

In this simple modeling exercise, we have set out the key steps that should be followed when creating a data model in DynamoDB. Write your entity diagram to identify the relationship between objects in your application. Carefully write down your access patterns and start modeling the structure of primary keys for your entities. Try to do your best with the main index, and then make the use of secondary indexes if needed. During the exercise, take a practical example and visualize it in the NoSQL Workbench.

Once done, celebrate, as scaling is not a problem anymore. In our example, a new book would represent a new partition, and therefore a new storage node: DynamoDB abstracts the complexity of database scaling from you. Because partitions are split up on different physical machines, and because the time to find the partition you query for is constant, your queries efficiency will remain stable as the amount of data increases.

However, this automatic scaling isn't free - the modelisation exercise is slightly more complicated than with relational database systems. You can't just go from your entities diagram to the data model. Access patterns must be thought of in advance, as they should drive the modelisation.

Because of this, there are a few situations in which DynamoDB might not be the optimal choice. If your application is meant to allow for a virtually unlimited number of access patterns - an analytics platform for instance - then DynamoDB will not be a good fit. Or, if you are building a new application from scratch, you may not know how its functionalities will look like in the longer run. It will then be tough to think of all the access patterns in advance. Finally, keep in mind the learning curve. If you plan to build a product as fast you can, and if your team is much more familiar with relational database systems, you might want to stick to it to maximize development speed.

If not in one of these situations, go and try DynamoDB. To speed up learning, I would strongly recommend reading The DynamoDB Book by Alex Debrie. You'll find plenty of well chosen examples as well as the main theoretical concepts behind it. Enjoy!

Icon Library in React: Why Inline SVG Are Better than a Font

Thu, 25 Mar 2021 00:00:00 GMT

How I managed to choose a performant icon system adapted to a Next.js project with styled-components.

Almost every website uses icons in order to quickly guide users or to improve the aesthetics of the page.

Usually, designers can export icons to SVG format. SVG stands for Scalable Vector Graphics which means they can grow indefinitely, as they are defined by a description, not by pixels like in pictures.

In a new Next.js/React project with a hundred icons, I had to choose the best way to integrate icons. In this article, I propose to:

Share a bench of four methods to integrate SVG
Compare the best promising methods: inline SVG and font
Detail the implementation of the retained method

So many ways to integrate SVG icons!

Once you have an SVG (let's say a wonderful arrow icon), there are at least four ways to import them... not all of these methods are worth it!

SVG as an image

An SVG can be used as the source of an img HTML tag:

const Icon = ({ src, ...imgProps }) => <img src={src} {...imgProps} />;

const ArrowIcon = <Icon src="icons/arrow.svg" alt="Big arrow" />;

Even if this icon can be manipulated exactly as an image, advantages of SVG, except file size reduction, cannot be used. In particular, customization is not possible. Moreover, if the file is moved or deleted, developers may not be aware of it. Let's see what's next!

SVG as a background image

A really simple way to create an icon is to set it as the background image of a div element.

import styled from "styled-components";

const Icon = styled.div`
  ${({ url }) => `background-image: url(${url});`}
  display: flex;
  align-items: center;
  justify-content: center;
  overflow: hidden;
  background-size: cover;
  background-position: center center;

  ${({ size }) => `
    width: ${size}px;
    height: ${size}px;
  `};
`;

const ArrowIcon = <Icon url="icons/arrow.svg" size={20} />;

The icon can be manipulated as a div element, so the size is customizable, but that's about it. Thus, let's go through another way.

SVG as font

One popular method to manage a big set of icons is to use a font. It is quite easy to generate a font with online tools such as IcoMooon or Fontello.

A generic icon component should look like:

import "./style.css";

const Icon = (props) => (
  <span>
    <i className={`icon-${props.slug}`} />
  </span>
);

Where the file style.css, normally created by a font generator, should be something like:

@font-face {
  font-family: "icons";
  src: url("fonts/icons.woff2") format("woff2"), url("fonts/icons.woff") format("woff");
  font-weight: normal;
  font-style: normal;
  font-display: block;
}

.icon-arrow:before {
  content: "\\e954"; /* `\\e954` is a character defined in the font */
}

For additional props, you can wrap the icon, replacing span by a styled component with props. For instance:

const WrappedIcon = styled.span`
  ${(props) => `
    i {
      ${props.color ? `color: ${props.color};` : ""}
      ${props.size ? `font-size: ${props.size}px;` : ""}
    }
  `};
`;

And finally get a simple icon component:

const ArrowIcon = <Icon slug="arrow" size={20} color="blue" />;

This method seems to be used in a lot of projects. Before challenging it, let's see the last one.

Inline SVG

Importing icons in React (granted it is properly configured, we'll see that later) is as simple as that:

import Arrow from "icons/arrow.svg";

const MyComponent = () => <Arrow />;

This is the method I chose in my case, let's understand why.

Comparison

Criteria ✅: OK ⚠️: Points of attention ⛔: KO if it is a regarded criterion	Inline SVG	Font	SVG as image	SVG as background image
Customization	✅ Highly customizable: `size`, `color`, etc. and specific properties `fill` and `stroke`	⚠️ Customizable like text only: `size`, `color`, etc.	⛔ Limited customization, same as image: `size`, etc.	⛔ Limited customization, via its div container: `size`, etc.
Customization	✅ All parts of the SVG can be customized finely	⚠️ Painful positioning as it uses text positioning CSS	⛔ Limited customization, same as image: `size`, etc.
Performance	✅ Smaller files compared to images	✅ Smaller files compared to images	✅ Smaller files compared to images	✅ Smaller files compared to images
	⚠️ Icons are loaded with the DOM: it may slow the page loading, but ensures all icons are loaded at the same time as the page	✅ Can be loaded thanks to a CDN	⚠️ Icons are loaded with the DOM: it may slow the page loading, but ensures all icons are loaded at the same time as the page	⚠️ Icons are loaded with the DOM: it may slow the page loading, but ensures all icons are loaded at the same time as the page
		⚠️ Font is loaded in parallel: blank characters can appear. All icons are loaded even if few are used
Maintainability	✅ Easy to add icons progressively	⚠️ Painful to regenerate a font, even if some external tools like Icomoon or Fontello can help	⛔ Difficult to assert images path are valid	⛔ Difficult to assert images path are valid
Developer Experience	✅ Good experience if SVG files are easy to customize	✅ Good experience once the font is configured	✅ Good experience	✅ Good experience
Accessibility	✅ Possibility to define `title` and `desc` tags	⛔ Not accessible without large efforts	✅ Accessible like images	⛔ Not accessible

Let's dive further into the comparison of the two concurrent methods: inline SVG vs. font...

Customization

Fonts are managed like text and it is therefore possible to customize an icon's size or color. Precise positioning is possible with the same properties as text (font-size, line-height, text-align, etc.), so it is quite painful when icons are used as buttons or images. For emojis used in a text, it could be more relevant, however.

SVG are customizable as images and properties fill and stroke allow to redefine colors respectively of the content or the border. All parts of the SVG can be separately customized, which is impossible to do with fonts.

➡️ Using inline SVG is the best choice if you need color variants of icons, or to customize precise parts of them.

➡️ Icons used as characters should be part of a font.

Performance

Fonts are loaded asynchronously, which means blank characters can appear while the font is being loaded.

All icons are loaded in the font, even if only a few of them are used on the page. However, it can be served by a CDN, a fast method if CORS is correctly configured.

On the contrary, SVG are loaded with the DOM, which avoids blank spaces, but relatively increases loading duration. An icon usually weighs from 500B to 5kB (less with SVG Optimization) so it can be non-negligible if hundreds of icons are used at a time - quite rare, however!

Fonts are also subject to anti-aliasing methods applied by browsers to font characters when scaling. SVG do not suffer from this issue and are fully scalable.

➡️ In pages with hundreds of icons, using a font reduces the page size. Serving it via a CDN is a performant option.

➡️ For pages with moderate use of icons, SVG is a better option to avoid anti-aliasing and blank characters at loading.

Maintainability

What if icons evolve along the development process?

With a font, you need to regenerate the font correctly, with the correct source files - otherwise, some character codes may change!

With SVG, you just need to add, replace or delete a file and update file imports in the project. In case of replacement, check the SVG structure is still the same, as fine customization of SVG parts may be affected.

➡️ If icons are likely to evolve, SVG files are a more consistent and independent format to handle icon addition, edition, or deletion.

Developer experience

Once icon systems are set, both methods are easy to use as developers, as seen in the first part.

The only point of attention can be when it is necessary to customize SVG icons that are already styled. For instance, with a fill or a width property already set:

<svg width="15px">
  <path fill="none" d="..." />
</svg>

In this case, applying custom fill or width won't be applied. Cleaning the SVG by removing fill="none" and width="15px" should fix the issue.

➡️ Once an icon system is set, the developer experience should be satisfying in both cases.

Browser compatibility

Fonts are supported by almost all browsers, using the @font-face rule. All modern web browsers support WOFF2 and WOFF which are optimized formats for the web, but other formats can be used for broader browser support. However, Opera Mini does not support @font-face at all.

All modern browsers now support inline SVG method, even Internet Explorer which only does not support CSS transforms (more details on caniuse website).

➡️ Both methods are supported by more than 98% of web browsers. Inline SVG is slightly better supported and has better fallback options.

Accessibility

Serving accessible fonts is hard. If you are adventurous, this article details the tricks to guarantee accessible fonts, so I won't expand on it here.

If accessibility is important for your project, then SVG are made for this. An SVG file may have a title or even a desc (for description) tags. For better browser support, it is possible to add an id to these tags and refer to them the attributesaria-labelledby as the following minimal example shows.

<svg aria-labelledby="circleTitle circleDescription" role="img">
  <title id="circleTitle">A circle</title>
  <desc id="circleDescription">A red circle with a blue border</desc>
  <circle cx="50" cy="50" r="40" stroke="blue" stroke-width="2" fill="red" />
</svg>

For more details, I advise you to look at this CSS-tricks page.

➡️ For accessible projects, SVG is the best option with no hesitation.

A winner?

A font system is an option to consider if your project uses a huge amount of icons per page, is focused on performance, does not need to be accessible, and has a graphic chart that won't evolve.

In other cases, for better flexibility and user experience, I would recommend an inline SVG system. That's what I chose in the case of a new B2C project with common use of icons (<30 per page) and a library with possible evolutions.

In the following part, I will explain how to configure such an icon system.

Configuring an SVG icon system with Next.js, React and styled-components

Here are the four simple steps I followed to integrate icons in a Next.js/React project using styled-components as styling library.

Install babel-plugin-inline-react-svg:

# With npm
npm install --save-dev babel-plugin-inline-react-svg
# or, with yarn
yarn add --dev babel-plugin-inline-react-svg

This package will allow the import of SVG files as React components. It also uses SVGO to optimize SVGs.

Note: SVGR also offers good alternatives to babel-plugin-inline-react-svg depending on your bundler, such as @svgr/webpack.

Add inline-react-svg plugin to babel configuration (babel.config.js or .babelrc):

plugins: ["inline-react-svg"];

For further configuration, refer to the npm page.

In a TypeScript project, you should also add a module declaration in a type file:

// fileTypes.d.ts

declare module "*.svg" {
  import React = require("react");
  const src: (props: React.SVGProps<SVGSVGElement>) => JSX.Element;
  export default src;
}

Now, it is possible to import directly SVG as React components. For a resilient icon system, it is possible to create a JSX/TSX file per icon:

// icons/Arrow.jsx
import Arrow from "./arrow.svg";

// ... (possible customization of the component)

export default Arrow;

With this method, refactoring icons is easier if the import method changes.

Warning: using a unique index file with all SVG icons gathered in it is not recommended, because it prevents from tree-shaking the library.

Import SVG files as React components:

import Arrow from "icons/Arrow";

const MyComponent = () => <Arrow />;

It is also possible to customize icons with styled-components, as shown in this example:

// MyComponent.tsx
import styled from "styled-components";
import Arrow from "icons/Arrow";

const CustomSpan = styled.span`
  color: red;
  font-size: 40px;
`;

const CustomArrow = styled(Arrow)`
  width: auto;
  height: 30px;
  stroke: blue;
  stroke-width: 50px;
  fill: currentColor; /* fill with the current color, here red */
`;

const MyComponent = () => (
  <button>
    <CustomSpan>
      Go <CustomArrow />
    </CustomSpan>
  </button>
);

...which renders as a wonderful button:

That's all you need to manage icons in your project!

File Download with Token Authentication and Javascript

Wed, 24 Mar 2021 00:00:00 GMT

Recently on a project of mine, I was presented with a problem: I needed to allow our users to download a ZIP archive of files from our back-end, based on complex authorisation rules, following a four-step process:

The front-end would call the back-end with a list of file names to download.
The back-end would authenticate the user with their JWT token.
It would then check if the user was allowed to retrieve those files.
Finally, it would send them the archive back.

Up until now, we were relying on ‘security through obscurity’, meaning we were trusting the uniquely generated file names to prevent users from snooping around and downloading files they were not supposed to retrieve.

But this was not enough: how could we revoke the rights of the user on a file? How could we be certain that a user would not be accessing a file they were not supposed to access? We needed to add authentication to the route.

The Unauthenticated Case

First, let me show you the code we were using until now. In this case, the back-end is a Spring Boot microservice, and the front-end is an Angular SPA.

Below is a simplified version of the unauthenticated case. What this does is define a GET route /files/download taking a set of file IDs as parameters, and retrieving, zipping and sending the files back to the client.

@GetMapping("/files/download")
public ResponseEntity<ByteArrayResource> download(
    @RequestParam Set<UUID> fileIds
) {
    ByteArrayResource zippedFiles = new ByteArrayResource(
        fileService.getZippedFiles(fileIds)
    );

    return ResponseEntity
        .ok()
        .contentLength(zippedFiles.contentLength())
        .header(HttpHeaders.CONTENT_TYPE, "application/zip")
        .header(
            HttpHeaders.CONTENT_DISPOSITION,
            ContentDisposition
                .builder("attachment")
                .filename("archive.zip")
                .build()
                .toString()
        )
        .body(zippedFiles)
    ;
}

So now, if you make a GET request to /files/download?fileIds=UUID, you’ll receive the file with name UUID zipped. This allows you to keep your front-end code extremely simple because a simple anchor element pointing to the URL to the back-end route will be enough to trigger a download pop-up in the client’s browser, thanks to the Content-Disposition header.

So your front-end component could retrieve the required file names, concatenate them somehow, and then simply display a link to the back-end URL, like so:

<a
  href="{{ backendUrl }}/files/download?fileIds={{ fileNames$ | async }}"
>Download your archive</a>

This is all good and well, but when you do this, you cannot forward JWT tokens, so you cannot authenticate your user: you need to download the files some other way.

Adding Authentication to the Route

So first, you need to add some kind of authentication and authorization. Though my project required complex authorization rules, let us say you only want to be sure the user is logged in before sending them the files. This is only a matter of adding a decorator to your back-end route:

@GetMapping("/files/download")
@PreAuthorize("hasRole(T(fr.theodo.security.Roles).USER)")
public ResponseEntity<ByteArrayResource> download(…) {…}

This is the easy part. Now, if you were to attempt using the aforementioned link, you would get an error: as no authentication token is provided, the back-end responds with a 401 Unauthorized response. How do we fix that?

Retrieving the File with JavaScript

Downloading the file will be done in two steps: first, you will download the file using JavaScript, allowing you to set the authentication token, then, you will ‘forward’ the file to your user.

Alternatively, note that you could set up a different authentication method specifically for this route, such as a cookie-based one, though this would mean having two different authentication methods on your project, with one being used by a single route… so I would advise against it.

Downloading the File

Assuming you already perform authenticated calls to your back-end using some kind of API client, downloading the file will be straightforward: you will instantiate your client in your component, and call your back-end (here, it is done with this.apiClient.downloadZipFile).

The switchMap RxJS operator allows you to discard any previous calls if this.project$ changes; you can read more about it on Learn RxJS.

export class MainPanelComponent {
  @Input() project$!: Observable<Project>;

  constructor(private apiClient: ApiClientService) {}

  private getZip(): Observable<string> {
    return this.project$.pipe(
      switchMap(project => this.apiClient.downloadZipFile(project.files)),
    );
  }
}

‘Forwarding’ the File to Your User

Now that you have retrieved your file, returned as an Observable by switchMap, you need to trigger the save on your user’s computer. This is only a matter of chaining two new operators in the RxJS pipe:

…
return this.project$.pipe(
  switchMap(project => this.apiClient.downloadZipFile(project.files)),
  map(data => window.URL.createObjectURL(data)),
  tap(url => {
    window.open(url, '_blank');
    window.URL.revokeObjectURL(url);
  })
);
…

First, you map the retrieved data to an object URL (a simple string). MDN will explain them in greater details, but the main things you need to know are that they are a way to create local URLs pointing to the browser memory… and that they are not supported by Internet Explorer, so you may want to keep that in mind depending on your target audience.

Finally, you tap into this new Observable to expose the file to your user (simply opening a new window with this local URL will work) and clean up behind yourself by removing the object URL (this allows the browser to free up some memory once the file is downloaded).

Triggering the Download

You may have noticed that for now, you have a getZip method, but you do not call it. To remedy this, you only need two new lines of code in your component and one in its template:

export class MainPanelComponent {
  @Input() project$!: Observable<Project>;

  zipDownloadTrigger = new EventEmitter<void>();

  constructor(private apiClient: ApiClientService) {
    this.zipDownloadTrigger.pipe(exhaustMap(this.getZip)).subscribe();
  }

  …
}

<button (click)="zipDownloadEmitter.emit()">Download your archive</button>

This defines an event emitter, which emits after each click on a button. You chain this emitter with an exhaustMap RxJS operator, which will call the getZip method, and wait for it to complete before resuming listening to the event emitter (hence the ‘exhaust’; again, refer to Learn RxJS for more details).

This means that all clicks on the button are ignored while the component is downloading the file. Note that you could do without this, in which case you could simply pass the getZip method as a click callback on the button.

Note that for the sake of simplicity, I use a button here, but this is a bad practice in terms of accessibility: you should always use a download link to download a file. Here you could have a button to trigger the download, then display the link with the local URL as its href attribute.

UX Niceties and the Final Code

If you put all this together and add a simple loading toggle, you get the following code:

export class MainPanelComponent {
  @Input() project$!: Observable<Project>;

  isTheArchiveLoading: boolean = false;
  zipDownloadTrigger = new EventEmitter<void>();

  constructor(private apiClient: ApiClientService) {
    this.zipDownloadTrigger.pipe(exhaustMap(this.getZip)).subscribe();
  }

  private getZip(): Observable<string> {
    this.isTheArchiveLoading = true;

    return this.project$.pipe(
      switchMap(project =>
        this.apiClient
          .downloadZipFile(project.files)
          .pipe(finalize(() => { this.isTheArchiveLoading = false; }))
      ),
      map(data => window.URL.createObjectURL(data)),
      tap(url => {
        window.open(url, '_blank');
        window.URL.revokeObjectURL(url);
      })
    );
  }
}

This ensures that you can show a loader to your user if isTheArchiveLoading is true. The finalize operator in the pipe on downloadZipFile is called when downloadZipFile resolves, much like the Promise.finally() method.

And here you go, you have a fully functional downloading method, with an authenticated route, called from your front-end!

Improve accessibility on your Angular Apps with Jasmine and Axe

Mon, 22 Mar 2021 00:00:00 GMT

At Theodo, we have multiple projects using Angular and we are currently focusing on building more accessible websites. (learn more about a11y)

We had an experience with Axe and jest-axe on other React projects, and we wanted something similar for Jasmine, the test framework on the Angular project.

What is Axe

Axe is an open source accessibility testing engine by Deque. It can automatically find about 30% of accessibility defects, such as missing labels on inputs, missing alt on images, ...

We found some tests examples with Jasmine in the Axe documentation but it did not seem as straightforward to use as jest-axe. So we decided to solve this problem and make an open source matcher for Jasmine

How to use jasmine-axe

Here is an example on how to use jasmine-axe:

import { TestBed } from '@angular/core/testing';
import { FormsModule } from '@angular/forms';
import { axe, toHaveNoViolations } from 'jasmine-axe';
import { FormComponent } from './form.component';

describe('FormComponent', () => {
  beforeEach(() => {
    TestBed.configureTestingModule({
      imports: [FormsModule],
      declarations: [FormComponent],
    });
    TestBed.compileComponents();
    jasmine.addMatchers(toHaveNoViolations);
  });

  it('should pass accessibility test', async () => {
    const fixture = TestBed.createComponent(FormComponent);

    expect(await axe(fixture.nativeElement)).toHaveNoViolations();
  });
});

Axe will find the violations and suggest solutions to fix them.

A violation was found in a component: Axe suggests some solutions to fix it.

We cannot rely only on those tests for ensuring full accessibility of a website, but it allows us to automate some checks and catch common errors.

We are eager for feedback so try it on your Angular project and tell us if it helped your team improve accessibility!

The React Native Guide I Wish I Had! - Part 1: Environment and Stateless Components

Mon, 22 Mar 2021 00:00:00 GMT

Quick Links

→ Part 1 - focuses on setting up your environment for react native, typescript and redux, as well as creating stateless components for a Todos app. (you are here!)

→ Part 2 - focuses on integrating redux and adding stateful components to your Todos app.

Table of Contents

Project Source
Introduction
Environment Setup
   Technologies
       Homebrew
       Node Version Manager
       Node & NPM
       React Native CLI
   Text Editor
The Todos App
   Creating The App
       Installing Icons
   Folder Structure
   Designing Components
       Styling
       Stateless Components
Resources

Project Source

For reference, the source code for this project can be viewed and downloaded here.

Introduction

During my first week at Theodo, I had a lot to learn about the main technologies used within the company. I focused my attention on learning to build a simple React Native application using Typescript and Redux. This post is aimed at those with a little knowledge of these technologies, guiding you from environment setup, to having a simple working Todos app! I should reiterate that some React, React Native, Typescript and Redux knowledge is assumed. Check out the Resources section for some useful beginner guides!

Environment Setup

The bane of a developer’s life is often related to setting up a working environment for your development. I’m going to outline the steps that I took to fully set up my development environment for this little project. I’ll also be writing this from a Mac OS perspective, but with the resources provided, I’m sure Unix/Windows translation won’t be too difficult!

Technologies

Homebrew

I try to install all the technologies that I need into a central location. This makes using homebrew - a very well maintained package manager for Mac OS - a natural choice.

To install, open a terminal and type:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Node Version Manager

Though this isn’t completely necessary, I like to install Node version manager so that I can have multiple easily manageable versions of Node installed on my Mac at once.

To install:

Open a terminal and type brew install nvm
Create NVMs working directory with mkdir ~/.nvm

Open (or create) your ~/.profile (or ~/.zprofile if using zsh instead of bash) and add:

export NVM_DIR="$HOME/.nvm"
[ -s "$(brew --prefix)/opt/nvm/nvm.sh" ] && . "$(brew --prefix)/opt/nvm/nvm.sh" # This loads nvm
[ -s "$(brew --prefix)/opt/nvm/etc/bash_completion.d/nvm" ] && . "$(brew --prefix)/opt/nvm/etc/bash_completion.d/nvm" # This loads nvm bash_completion

Now every time you log in to your mac, your .profile (or .zprofile) will load NVM and its bash (or zsh) completion. Instead of logging out and logging in, we can invoke this file by running (replace .profile with .zprofile if using zsh) . ~/.profile in your terminal.

Make sure that NVM has been installed properly by typing nvm -v in your terminal.

Note: You can also place this last section in .bashrc (or .zshrc), but .bashrc (or .zshrc) is invoked every time a terminal window is opened. We only need to load NVM once, so we may as well just do it when we login by placing the code in .profile (or .zprofile)!

Node & NPM

We can install Node and NPM using NVM from the last step. Simply run these two commands in a terminal to install and use the latest version of Node (comes with NPM):

nvm install node
nvm use node
Check the installation worked using node -v in a terminal window.

React Native CLI

React Native command-line interface makes it easy to create a new React Native application.

Head over to the React Native setup guide.
Select the ‘React Native CLI Quickstart’ tab.
Do not install Node using homebrew (you should already have it through NVM)!
Install watchman using homebrew.
Follow the ‘Installing dependencies’ steps for both Target OS’s.

Text Editor

You can use whichever text editor you like, but I am using Visual Studio Code, which I highly recommend as it's very customisable!

Here is a list of the extensions that I use:

Many of these are configurable and there are lots of resources online for customisation!

The Todos App

Now that we have all of the necessary technologies installed, let’s make a React Native todos app!

Creating The App

Using the React Native CLI, we can easily create a React Native app from our terminal. Navigate to a directory where you’d like to store the app, then run:

npx react-native init *AwesomeTSProject* --template react-native-template-typescript

You can replace AwesomeTSProject with whatever project name you like. We are also using the Typescript template for our React Native app.

Note: npx will access the current version of React Native CLI at runtime so that you don’t have to install and manage a global version!

Next, navigate into your project directory and run Metro, a javascript bundler that ships with react native:

cd AwesomeTSProject
npx react-native start

Then run your application on iOS (or for android, replace run-ios with run-android):

npx react-native run-ios

Note: React Native version 0.63.4 is unable to install the most recent version of a debugging tool called Flipper on iOS. If you are running into errors involving this module, navigate to /ios/Podfile and replace use_flipper! with use_flipper!({ 'Flipper-Folly' => '2.3.0' }). Then delete the Podfile.lock and, in a terminal, navigate to the /ios directory and run pod install. Now it should be fixed!

Installing Icons

In order to use icon fonts in a React Native app, they must be installed. I use react-native-vector-icons from NPM.

Android Installation

Go here and follow the recommended installation instructions.

iOS Installation

Open Xcode and open the ProjectName/ios/ProjectName.xcodeproj file.
Follow these (manual) instructions
You may need to run pod install in the ios directory before restarting your application.

Folder Structure

I have created a src directory containing all of the source code for my application. The first step is simply to move App.tsx inside this directory and update the import in index.js.

Each of the remaining folders inside src are as follows:

components - all the React components that build up a page live inside this folder (e.g. custom buttons and inputs.
config - all the files for configuration live here (e.g. default styles, colours)
screens - all the screens for the application live here. They are React components that compose many components from the components directory. (Note: there’s only one screen in this application)
store - all things Redux are stored here. Each folder within store denotes a feature. Each feature folder can contain types, actions, selectors and reducers.

I urge you to create these folders now! Don’t worry about the files, we’ll get to those later.

Designing Components

First, I like to think about the smallest components that I’ll need in my application and then build up. We’ll start with a text component.

Although React Native already supplies a Text component, I like to create an AppText component that encapsulates the default styles for my text. This way, we don’t have to repeat any text styling. Simply create a new AppText.tsx file in the components folder and look at the following code.

import React, { FunctionComponent } from "react";
import { Text, TextStyle } from "react-native";

import defaultStyles from "../config/styles";

interface Props {
  style?: TextStyle;
}

const AppText: FunctionComponent<Props> = ({ children, style }) => {
  return <Text style={[defaultStyles.text, style]}>{children}</Text>;
};

export default AppText;

/src/components/AppText.tsx

Just like in React, we create a functional AppText component, passing in its destructured props. Notice that props are typed using an interface (or type) containing all of the passed props. This interface (or type) is then passed to the generic type of the function component. The children prop comes from the FunctionComponent itself. We can also pass optional styles as a prop when using the AppText component, which has the same type as styles that you'd apply on a React Native Text component.

Styling

Styling a component works by setting its style prop to a style object. This can be inline, but it is good practice to create a stylesheet where each key is a separate style object. The style prop can also be given an array of style objects, where the last style object will take precedence if there are any conflicting rules (much like CSS!).

Any component that can be styled, follows the Flexbox styling routine.

Notice that we are using are defaultStyles.text as the first style for the text component. This is imported from ../config/styles. Let’s go and create that file now.

import { Platform, StyleSheet } from "react-native";

import colors from "./colors";

export default StyleSheet.create({
  text: {
    color: colors.dark,
    fontSize: 18,
    fontFamily: Platform.OS === "android" ? "Roboto" : "Avenir",
    fontWeight: "700",
  },
});

/src/config/styles.ts

This file simply exports a StyleSheet object, which can be imported anywhere in the app! Notice how it uses the colors module. This is another configuration file which is just an object storing your colour names.

const colors = {
  black: "#000",
  white: "#fff",
  red: "#de6e66",
  green: "#8bc989",
  light: "#f0f0f0",
  medium: "#6b6b6b",
  dark: "#0c0c0c",
};

export default colors;

/src/config/colors.ts

Now we can test our component works by navigating to src/App.tsx and replacing the code with the following.

import React, { FC } from "react";
import { SafeAreaView, StyleSheet } from "react-native";

import AppText from "./components/AppText";
import colors from "./config/colors";

const App: FC = () => {
  return (
    <SafeAreaView style={styles.container}>
      <AppText style={styles.text} />
    </SafeAreaView>
  );
};

const styles = StyleSheet.create({
  container: {
    alignItems: "center",
    backgroundColor: colors.light,
    flex: 1,
    padding: 20,
  },
  text: {
    color: colors.red,
  },
});

/src/App.tsx

result

Stateless Components

Now that you know the basics of building stateless components, try building these components yourself. Take a look at my source code if you get stuck!

Button.tsx

AppTextInput.tsx

Card.tsx

Before we move on to Redux, it’s worth taking a look at the /src/components/Screen.tsx component. This component encapsulates some logic to ensure that all content in the application is within the safe area of the screen (not underneath the notch). It also adds padding equal to the height of the status bar if we are using Android. We will use this component in the top level of our todos screen in the screens directory.

Part 2: Redux and Stateful Components

Now you're ready to move on to looking at integrating Redux into your Todos app, as well as adding some stateful components!

Click here for Part 2!

Resources

React Tutorial - useful for learning React!
React Hooks - if you don’t know what react hooks are or how to use them, the official docs are great!
React Native Basics - useful for learning React Native!
Redux Essentials and Redux with React - both on the redux official docs, great source of information for redux newbies!
Typescript in 5 mins! - learn the basics of Typescript!

The React Native Guide I Wish I Had! - Part 2: Redux and Stateful Components

Mon, 22 Mar 2021 00:00:00 GMT

This article takes you through a crash course of redux implementation, as well as adding it to your Todos application.

Quick Links

→ Part 1 - focuses on setting up your environment for react native, typescript and redux, as well as creating stateless components for a Todos app.

→ Part 2 - focuses on integrating redux and adding stateful components to your Todos app. (you are here!)

Table of Contents

Redux
   Actions
   Reducers
   Selectors
   Configure and Connect The Store
Bringing It All Together
   Stateful Components
       AddTodo
       TodosList
Wrapping Up
Resources

Redux

Note: Redux Toolkit is now the recommended way of implementing Redux logic. It wraps the core Redux logic, reducing boilerplate code and potential mistakes. This guide doesn't use Redux Toolkit, but gives a thorough introduction to Redux concepts.

I’ll explain redux in the context of how this application has been built. If you have not followed through Part 1 of this guide, it might be easier to have a look there first!

To give an overview - redux maintains a single state that can be updated using event-like actions and reducers, and accessed using selectors.

Actions

We’ll start with the actions that can be dispatched throughout the application (disregarding the all, complete and incomplete filters for now). An action is an object that has a type member and a payload member. The type is what we want to do (e.g. ADD_TODO or DELETE_TODO). The payload can be anything we want (e.g. the todo text, todo id).

With this knowledge, lets create a file in /src/store/todos called types.ts. Here we will define all the possible action types for a todo along with the todo type itself.

export const ADD_TODO = 'ADD_TODO';
export const DELETE_TODO = 'DELETE_TODO';
export const TOGGLE_TODO = 'TOGGLE_TODO';

export interface Todo {
  id: number;
  text: string;
  completed: boolean;
}

interface AddTodoAction {
  type: typeof ADD_TODO
  payload: string
}

interface DeleteTodoAction {
  type: typeof DELETE_TODO
  payload: number
}

interface ToggleTodoAction {
  type: typeof TOGGLE_TODO
  payload: number
}

export type TodoActionTypes = AddTodoAction | DeleteTodoAction | ToggleTodoAction;

/src/store/todos/types.ts

A todo is defined as an object with an id, some text and a completed status.

Each todo action is defined with its respective type and payload (add todo’s payload is the todo text, delete and toggle todo’s payload is the todo id).

Finally we export a new type which is a composition of all the toto action interfaces.

Now we can move on to creating the todo actions.

import { ADD_TODO, DELETE_TODO, TOGGLE_TODO, TodoActionTypes } from './types';

export const addTodo = (todo: string): TodoActionTypes => (
  { type: ADD_TODO, payload: todo }
);

export const deleteTodo = (id: number): TodoActionTypes => (
  { type: DELETE_TODO, payload: id }
);

export const toggleTodo = (id: number): TodoActionTypes =>  (
  { type: TOGGLE_TODO, payload: id }
);

/src/store/todos/actions.ts

Later on, we will dispatch these todo action creators in the necessary components, much like a Javascript event. Each one is simply a function that takes in the necessary information (todo text, todo id) and returns an action object (type and payload). These actions are consumed by a reducer.

Reducers

Reducers are functions that take a state and an action, perform updates on the state and return a new state. These are much like Javascript event listeners (see the connection between actions - dispatchable events - and reducers - event listeners?).

import { ADD_TODO, DELETE_TODO, TOGGLE_TODO, Todo, TodoActionTypes } from './types';

const initialState: Todo[] = [];

let id = 0;

const todosReducer = (state = initialState, action: TodoActionTypes): Todo[] => {
  switch (action.type) {
  case ADD_TODO:
    return [
      {
        id: ++id,
        text: action.payload,
        completed: false,
      },
      ...state,
    ];
  case DELETE_TODO:
    return state.filter(todo => todo.id !== action.payload);
  case TOGGLE_TODO:
    return state.map((todo) => {
      if (todo.id !== action.payload) {
        return todo;
      }
      return {
        ...todo,
        completed: !todo.completed,
      };
    });
  default:
    return state;
  }
};

export default todosReducer;

/src/store/todos/reducer.ts

A reducer can seem very daunting at first, but it’s not so difficult! Let’s work through it.

It’s good practice to always create an initial state. For our todos reducer, the initial state is an empty list of todos.
I also create an id outside of the reducer. This is to be used inside the reducer to auto increment the id of a newly created todo.
The reducer takes two arguments - the current state and the action.

Inside the body of the function, we look at the action type and perform the necessary updates. In the case of ADD_TODO, we return the original state (list of todos) with the new todo prepended.

Selectors

Selectors, as they sound, are functions that extract refined pieces of information from the global state. They are useful to avoid writing duplicate code and integrate very nicely with react hooks!

import { RootState } from '../types';
import { Todo } from './types';

export const selectTodos = (state: RootState): Todo[] => state.todos;

export const selectTodoIds = (state: RootState): number[] => (
  state.todos.map(todo => todo.id)
);

export const selectTodoById = (state: RootState, id: number): Todo | undefined => (
  state.todos.find(todo => todo.id === id)
);

/src/store/todos/selectors.ts

Each selector derives information from the root state. We will define this type shortly, but as you can see, selectTodos function simply takes the todos list from the root state. selectTodoIds maps each todo to its id and returns an array of ids. selectTodoById searches the todos list and returns a todo with a specified id.

Configure and Connect the Store

Now that we have all the building blocks for maintaining todo state, we should create and connect our store to our application. First, let’s define a type for the root state. This type is the definition for our entire state. So if we added another feature, user authentication for example, we would add this as another key in our RootState type.

import { Todo } from './todos/types';

export interface RootState {
  todos: Todo[];
}

/src/store/types.ts

Then we create our store configuration file.

import { Store, combineReducers, createStore } from 'redux';

import { RootState } from './types';
import todosReducer from './todos/reducer';

const rootReducer = combineReducers<RootState>({
  todos: todosReducer,
});

const configureStore = (): Store => {
  return createStore(rootReducer);
};

export default configureStore;

/src/store/configureStore.ts

First we create a root reducer by combining all other reducers using redux’s combineReducers function. For now, this is only the todos reducer. We’ll give our todos reducer a key of todos (as our todos reducer state is a list of todos). Note that we specify the type of our root reducer by passing it as a generic type to combineReducers.

Finally we export a function that returns a redux store. This function makes use of redux’s createStore function, called with our root reducer.

You might ask why we don’t export the invoked createStore itself? This is because we only want to create the store when the application starts, so we export a function to be invoked when the app's root view is rendered.

Bringing It All Together

Navigate back to the /src/App.tsx file. Now we can import our configureStore function and invoke it before we render our application.

import React, {FC} from 'react';
import {StyleSheet, View} from 'react-native';

import {Provider} from 'react-redux';
import TodosScreen from './screens/TodosScreen';
import colors from './config/colors';
import configureStore from './store/configureStore';

const store = configureStore();

const App: FC = () => {
  return (
    <Provider store={store}>
      <View style={styles.container}>
        <TodosScreen />
      </View>
    </Provider>
  );
};

const styles = StyleSheet.create({
  container: {
    backgroundColor: colors.light,
    flex: 1,
  },
});

export default App;

/src/App.ts

I appreciate that we don’t have the TodosScreen component just yet, but that can be ignored for now. Our main focus is the new Provider component. This is a component provided by redux that allows us to pass down the store behind the scenes, making it accessible by any of its children. We pass the store into the Provider through its props.

Stateful Components

There are 4 components left to make.

AddTodo.tsx

TodoItem.tsx

TodosList.tsx

TodosScreen.tsx

We’ll work through AddTodo together, then I’ll let you do the rest! You should create them in this order:

AddTodo
TodoItem
TodosList
TodosScreen (make this component in the /src/screens directory)

AddTodo

AddTodo is a standard react function component.

import React, {useState} from 'react';

import AppTextInput from './AppTextInput';
import Button from './Button';
import Card from './Card';
import {addTodo} from '../store/todos/actions';
import {useDispatch} from 'react-redux';

const AddTodo: React.FunctionComponent = () => {
  const [todoText, setTodoText] = useState('');
  const dispatch = useDispatch();

  const handleSubmit = () => {
    if (!todoText) return;

    dispatch(addTodo(todoText));
    setTodoText('');
  };

  return (
    <Card title="Add New Todo">
      <AppTextInput
        onSubmitEditing={handleSubmit}
        icon="format-list-bulleted-type"
        placeholder="New Todo"
        onChangeText={(text) => setTodoText(text)}
        value={todoText}
      />
      <Button title="Add Todo" onPress={handleSubmit} />
    </Card>
  );
};

export default AddTodo;

/src/components/AddTodo.tsx

We use the react useState hook to maintain a local state of the todo text entered in the input field. See on AppTextInput, the onChangeText prop sets the todoText value, and the todoText is the value of the input itself (this is called two way binding on the todoText state variable).
Next we use the useDispatch hook from react-redux. This hook returns a reference to the dispatch function from the redux store. We can use this to dispatch any action as needed.
Our handleSubmit function abstracts the logic for submitting a todo. If there is no text, we do nothing, otherwise we use our dispatch reference to dispatch an addTodo action creator with our todo text. Finally we set the local todoText state variable to an empty string in order to clear the input box.
We use the handleSubmit function on press of the submit button, as well as onSubmitEditing of the text input (this handles adding a todo when return is pressed on the device keyboard).

That’s it! There aren’t many extra pieces to add in order to create a stateful react component with redux. I’ll now let you build the other 3 components. Don’t forget to use the useSelector hook with your todo selectors! The hook will return whatever your selector does.

TodosList

I should mention that the final version of my project includes todo filters, which haven’t been spoken about here. These filters largely affect this component, so here is the code minus the filter logic.

import {FlatList, StyleSheet} from 'react-native';
import {shallowEqual, useSelector} from 'react-redux';

import React from 'react';
import TodoItem from './TodoItem';
import {selectTodoIds} from '../store/todos/selectors';

const TodosList: React.FC = () => {
  const todoIds = useSelector(selectTodoIds, shallowEqual);

  return (
    <FlatList
      data={todoIds}
      keyExtractor={(id) => id.toString()}
      renderItem={({item}) => <TodoItem id={item} />}
      style={styles.container}
    />
  );
};

const styles = StyleSheet.create({
  container: {
    width: '100%',
  },
});

export default TodosList;

/src/components/TodosList.tsx

There is an optimisation in this component worth talking about. You might ask why we pass a list of ids instead of a list of todos. The reason for this is so that we can use the shallowEqual function as a second argument to our useSelector hook.

The second parameter of the useSelector hook is a comparator. This comparator is used to compare items from a new state, to items from the previous state. If the item has not changed, then redux will not trigger a re-render of this item.

If we did not use this comparator, then every time we update a todo’s ‘completed’ status, the whole list of todos would re-render!

You can also look at using the [useMemo](https://reactjs.org/docs/hooks-reference.html#usememo) hook as an alternative to using a comparator.

Wrapping Up

In this guide, I haven’t included the addition of the todo status filters (all, complete, incomplete), but you should now have the tools to build this yourself.

The steps you need to take for this are:

Create a /src/store/filters directory.
Create the necessary types, actions, reducer and selectors.

(Hint: you'll need a new todos selector that selects todos based on the current filter!)
Create the FilterButton component (Hint: it should dispatch a CHANGE_FILTER action creator!).
Add some FilterButton components to either the TodosScreen or TodosList component.

The source code on my github repository has this completely implemented if you need help or inspiration

⭐️ Thanks for reading! ⭐️

There are many, many more pieces to learn about React Native, Redux and application architecture, like: testing, routing, native features, making api calls, asynchronous state updates (Redux Saga/Thunk).

Resources

React Tutorial - useful for learning React!
React Hooks - if you don’t know what react hooks are or how to use them, the official docs are great!
React Native Basics - useful for learning React Native!
Redux Essentials, Redux - Getting Started and Redux with React - all on the redux official docs, great source of information for redux newbies!
Typescript in 5 mins! - learn the basics of Typescript!

Using Native Incoming Call UI for your React Native App (How I Wrote My First Native Module)

Wed, 17 Mar 2021 00:00:00 GMT

A story about my struggles implementing native call display functionality and how this led me to writting my own native module for android. Suitable for developers who are interested in how to get started with native modules for android within React Native or implementing native UI for incoming calls.

TL;DR

A project I was working on required native call UI to be implemented so that when users received a call notification from firebase their phone would ring as if they were receiving a native call. This was a really interesting feature to build but it did not come without difficulty. The library react-native-callkeep was used to implement this solution, this library acts as a fully managed connection service for implementing call UI using react native. However, the project had its own in-call UI we wanted to use, therefore we only wanted to use the incoming call functionality of callkeep. This solution seemed to work fine for iOS with the base recommended implementation, with the downside that it didn't work as smoothly for the latest android versions we wanted to run (android with a minimum api of 27). To get android working properly for these versions I had to write a native module which mimicked the backToForeground function from react-native-callkeep.

Native call UI trying to be implemented:

Intro

When I first realised the project I was working on would need an incoming call display that transitions into a React Native app I (foolishly) didn't think much of it, and assumed a simple library would solve most of my worries. As a team we decided to use the react-native-callkeep library which seemed to have all of the functionality required, which was to be able to display incoming call UI after a notification is received.

This implementation however, started to show a few issues when testing functionality on the latest android versions. These issues then led me down a track which ended with implementing a native android module. Writing this native module eventually did result in the desired functionality, although, to properly explain my decision into doing this I need to start from the beginning.

Using react-native-callkeep

The base implementation of callkeep is pretty straight forward and works well out of the box, when initially experimenting with it, the only problem I ran into was that it wouldn't display the native incoming call UI on certain android emulators. Switching from an android 11 emulator to an android 10Q emulator seemed to do the trick (note: this was only a problem for the emulator as the same functionality seemed to work on a real android 11 phone).

After following the basic installation guide (Found here), I would recommend setting up a callkeep class in your project. The basics to get the display incoming call UI to work for my project was:

import RNCallKeep, { AnswerCallPayload } from 'react-native-callkeep';

type DidDisplayIncomingCallArgs = {
  error: string;
  callUUID: string;
  handle: string;
  localizedCallerName: string;
  hasVideo: string;
  fromPushKit: string;
  payload: string;
};

export class CallKeep {
  private static instance: CallKeep;
  private callId: string;
  private callerName: string;
  private callerId: string;
  private isAudioCall: string | undefined;
  private deviceOS: string;
  private endCallCallback: Function | undefined;
  private muteCallback: Function | undefined;
  public IsRinging: boolean = false;

  constructor(
    callId: string,
    callerName: string,
    callerId: string,
    deviceOS: string,
    isAudioCall?: string
  ) {
    this.callId = callId;
    this.callerName = callerName;
    this.callerId = callerId;
    this.isAudioCall = isAudioCall;
    this.deviceOS = deviceOS;

    CallKeep.instance = this;
    this.setupEventListeners();
  }

  public static getInstance(): CallKeep {
    return CallKeep.instance;
  }

  endCall = () => {

    RNCallKeep.endCall(this.callId);

    if (this.endCallCallback) {
      this.endCallCallback();
    }

    this.removeEventListeners();
  };

  displayCallAndroid = () => {
    this.IsRinging = true;
    RNCallKeep.displayIncomingCall(
      this.callId,
      this.callerName,
      this.callerName,
      'generic'
    );
    setTimeout(() => {
      if (this.IsRinging) {
        this.IsRinging = false;
        // 6 = MissedCall
        // https://github.com/react-native-webrtc/react-native-callkeep#constants
        RNCallKeep.reportEndCallWithUUID(this.callId, 6);
      }
    }, 15000);
  };

  answerCall = ({ callUUID }: AnswerCallPayload) => {

    this.IsRinging = false;

    navigate('somewhere'); // navigated to call screen in our app
  };

  didDisplayIncomingCall = (args: DidDisplayIncomingCallArgs) => {
    if (args.error) {
      logError({
        message: `Callkeep didDisplayIncomingCall error: ${args.error}`,
      });
    }

    this.IsRinging = true;
    RNCallKeep.updateDisplay(
      this.callId,
      `${this.callerName}`,
      this.callerId
    );

    setTimeout(() => {
      if (this.IsRinging) {
        this.IsRinging = false;
        // 6 = MissedCall
        // https://github.com/react-native-webrtc/react-native-callkeep#constants
        RNCallKeep.reportEndCallWithUUID(this.callId, 6);
      }
    }, 15000);
  };

  private setupEventListeners() {
    RNCallKeep.addEventListener('endCall', this.endCall);
    RNCallKeep.addEventListener(
      'didDisplayIncomingCall',
      this.didDisplayIncomingCall
    );

  private removeEventListeners() {
    RNCallKeep.removeEventListener('endCall');
    RNCallKeep.removeEventListener('didDisplayIncomingCall');
    this.endCallCallback = undefined;
  }
}

This setup will get you the basics that you will need to handle displaying the incoming call UI along with a base setup for answering and ending calls. The key functions here to get only the incoming call UI working is the RNCallKeep.displayIncomingCall and RNCallKeep.endCall.

Initial problems with android

After my initial experiments with callkeep it seemed that IOS worked fine out of the box, however, there were some difficulties to get the same functionality working for android. The difficulties I came across with android are summarised below:

Emulator was not displaying the incoming call UI when correct command was called (RNCallKeep.displayIncomingCall)
- After experimenting with different emulators and getting the functionality to work, this led me to the conclusion that the android 11 emulator was not set up correctly to use this library.
- Changing to an android 10Q api 29 android fixed this issue, and the incoming call UI was successfully displayed
Answering the call from the incoming UI display on android was not navigating me to the app, the UI acted as if I had never answered the call.
- Solving this issue required some experimenting, but in the end was solved by calling RNCallKeep.endCall on the callAnswered listener and then navigating to the correct page on our app.
- Calling the RNCallKeep.endCall seemed to prompt android to stop displaying the UI and run the rest of the commands in the answered call listener function
- This step needed to be done on android only.
When a call was received, if android was locked, the user would not be navigated to the app but would instead be stuck at their home screen.
- The functionality required here instead of this issue would be for the user to be prompted to unlock their phone, and then be navigated to the app.
- This condition was a bit trickier and required me to dive into the realms of native app development as discussed below.

Writing a native module for android

Fulfilling the functionality required by the lockscreen took some further investigation, the callkeep library instructs you to call the android only function RNCallKeep.backToForeground to bring the react-native app to the foreground. This seemed to work while the app was in background mode but not while the app was in a lockscreen state. Looking into this function I found that it was using some java functions that were deprecated for the android versions necessary for our project. This led me to recreate this backToForeground function natively with an updated set of java functions that would work for the required versions.

Since I am working in React Native to write native code and call it from typescript/javascript I needed to use the React Native bridge. The android bridging documentation from React Native (Found here) has all of the necessary steps required to set up the java module and call it from within your app. Below is an example of the java module I created to start the app activity and request the user to unlock their phone.

package com.ExampleProject;

import android.app.Activity;
import android.app.ActivityManager;
import android.app.KeyguardManager;
import android.content.Context;
import android.content.Intent;
import android.net.Uri;
import android.os.Build;
import android.os.Bundle;
import android.util.Log;
import android.view.WindowManager;

import androidx.annotation.RequiresApi;

import com.facebook.react.bridge.Arguments;
import com.facebook.react.bridge.NativeModule;
import com.facebook.react.bridge.Promise;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.bridge.ReactContext;
import com.facebook.react.bridge.ReactContextBaseJavaModule;
import com.facebook.react.bridge.ReactMethod;
import com.facebook.react.bridge.ReadableMap;
import com.facebook.react.modules.core.DeviceEventManagerModule;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class CallkeepHelperModule extends ReactContextBaseJavaModule {

  private static ReactApplicationContext reactContext;

  CallkeepHelperModule(ReactApplicationContext context) {
    super(context);
    reactContext = context;
  }

  @Override
  public String getName() {
    return "CallkeepHelperModule";
  }

  @RequiresApi(api = Build.VERSION_CODES.O_MR1)
  @ReactMethod
  public void dismissKeyguard(Activity activity){
    KeyguardManager keyguardManager = (KeyguardManager) reactContext.getSystemService(
      Context.KEYGUARD_SERVICE
    );
    boolean isLocked = keyguardManager.isKeyguardLocked();
    if (isLocked) {
      Log.d("CallkeepHelperModule", "lockscreen");
      keyguardManager.requestDismissKeyguard(
        activity,
        new KeyguardManager.KeyguardDismissCallback() {
          @Override
          public void onDismissError() {
            Log.d("CallkeepHelperModule", "onDismissError");
          }

          @Override
          public void onDismissSucceeded() {
            Log.d("CallkeepHelperModule", "onDismissSucceeded");
          }

          @Override
          public void onDismissCancelled() {
            Log.d("CallkeepHelperModule", "onDismissCancelled");
          }
        }
      );
    } else {
      Log.d("CallkeepHelperModule", "unlocked");
    }
  }

  @ReactMethod
  public void startActivity() {
    Log.d("CallkeepHelperModule", "start activity");
    Context context = getAppContext();
    String packageName = context.getApplicationContext().getPackageName();
    Intent focusIntent = context.getPackageManager().getLaunchIntentForPackage(packageName).cloneFilter();
    Activity activity = getCurrentActivity();
    boolean isRunning = activity != null;

    if(isRunning){
      Log.d("CallkeepHelperModule", "activity is running");
      focusIntent.addFlags(Intent.FLAG_ACTIVITY_REORDER_TO_FRONT);
      activity.startActivity(focusIntent);
      dismissKeyguard(activity);
    } else {
      Log.d("CallkeepHelperModule", "activity is not running, starting activity");
      focusIntent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK + Intent.FLAG_ACTIVITY_REORDER_TO_FRONT);
      context.startActivity(focusIntent);
    }
  }

  private Context getAppContext() {
    return this.reactContext.getApplicationContext();
  }
}

The main function of this module is startActivity, which when called gets the context for the React app, if the app is running the function brings the app to the foreground and checks if the keyguard needs to be dismissed, otherwise if the app is not running this function starts the apps activity as a new task. The keyguard is then dismissed by calling the dismissKeyguard function which uses the KeyguardManager to check if the android is locked or unlocked, and asks the user to dismiss the keyguard if the phone is locked.

Putting all of these steps together I could then call the native module in my react app whenever a call was accepted from the incoming UI using the answerCall listener function like so:

import { NativeModules } from "react-native";
import RNCallKeep from "react-native-callkeep";

answerCall = ({ callUUID }: AnswerCallPayload) => {
  if (this.deviceOS === "android") {
    const { CallkeepHelperModule } = NativeModules;
    CallkeepHelperModule.startActivity();
    RNCallKeep.endCall(this.callId);
  }
  this.IsRinging = false;
  navigate("somewhere");
};

Conclusion

Callkeep was able to be implemented as an incoming UI solution only, in both IOS and android platforms. On android, native code was written so that answering the app from the lockscreen would start the react native app and navigate you to the correct screen. After these adjustments the React Native callkeep library ended up being a great fit for this project, performing all the required functionality. I hope this article is able to help anyone deciding between solutions to display incoming call UI, and hopefully cut down on some research and debugging time. I will finally leave you with a gif of the working functionality in all its glory, happy coding!

How to easily handle admin authentication and permissions in Django with Okta

Tue, 09 Mar 2021 00:00:00 GMT

In my project, we are developing an admin interface to create and share documents to users who can see them and download them from the website we are creating.

However, the administration platform is dedicated to multiple teams from various countries. We don’t want them to be able to see or interact with each other's documents. Besides, those teams are meant to change, and they should manage themselves. For each team, we want to create a super admin account able to create, delete and change permissions for its team members only.

I will show how we handled this challenge with an effortless and fast solution.

TL;DR:

In this article, I will developp three main points:

Why use Okta to manage users and permissions, especially in Django admin?
How to add single sign-on connection using the mozilla-django-oidc library?
How to manage user permissions in Django from Okta ?

Why use Okta with Django admin?

One of Django's valuable features is the management of permissions for various users. Though, it is not always simple to separate users into distinct independent groups and to allow some of them to manage only a fraction of it without seeing the other users. Besides, we wanted a system allowing the admins to connect on the Django platform with their admin account.

For our application, the authentication and user management system must:

be secure (standard auth flows, follows security best practices, possible to implement two-factor authentication)
be fast and scalable.
be easy to work with from a developer standpoint.
be simple enough for the team managers.

To handle those requirements, we decided to use Okta tools. Okta is a company founded in 2009 providing a cloud software. This software allows you to manage and secure user identity and access management. Okta’s CEO announced they had over 100 million registered users in 2019. Besides, Okta is top-notch on up-to-date security measures and auth flows.

Okta main functionalities

As seen in the graph below, Okta offers four major functionalities:

A fully automated onboarding sending emails to users to help them create their account.
Identity access and management to provide group management and two-factor authentication.
Accesses are given in real-time when adding users and changing groups.
Okta cares about security and has a protocol in place in case of an incident.

In Okta, it is simple for an admin with the correct rights to create other users and grant them the right groups. This way, at each level of the organization, users and their permissions are manageable effortlessly.

Example of a user created in Okta with its groups

I will now explain how we integrated Okta with Django admin and how we used it to manage our users’ permissions.

Implement authentication workflow to Django through Okta

To add the admin connection with Okta to Django admin, our team used mozilla-django-oidc library. “A lightweight authentication and access management library for integration with OpenID Connect enabled authentication services.” OpenId Connect, also known as OIDC, is an identity layer on top of the OAuth 2.0 protocol. The End-User identity can be verified with the authentication thanks to an Authorization server. OpenID Connect introduces the concept of an id_token, which is a security token allowing the client to verify the user's identity. The id_token also gets basic profile information about the user. It introduces the UserInfo endpoint, an API returning information about the user.

The Sign-in protocol used follows the steps described in the next diagram.

Sign-in protocol diagram

We configured the callback url with the client_id and added the client_secret in our settings to acquire the token from Okta as written in the library documentation. In our case, we used a RS256 algorithm.

# In settings.py

# Okta
OKTA_DOMAIN = os.environ.get("OKTA_DOMAIN")
OKTA_TOKEN = os.environ.get("OKTA_TOKEN")

# User information
USER_CRM_ID = None
USER_EMAIL = None

# Okta Admin
OKTA_ADMIN_DOMAIN = OKTA_DOMAIN
OIDC_RP_SIGN_ALGO = "RS256"
OIDC_OP_JWKS_ENDPOINT = f"{OKTA_ADMIN_DOMAIN}/oauth2/v1/keys"
OIDC_OP_AUTHORIZATION_ENDPOINT = f"{OKTA_ADMIN_DOMAIN}/oauth2/v1/authorize"
OIDC_OP_TOKEN_ENDPOINT = f"{OKTA_ADMIN_DOMAIN}/oauth2/v1/token"
OIDC_OP_USER_ENDPOINT = f"{OKTA_ADMIN_DOMAIN}/oauth2/v1/userinfo"
OIDC_RP_SCOPES = "openid profile email groups"
OIDC_RP_CLIENT_ID = os.environ.get("OKTA_ADMIN_CLIENT_ID")
OIDC_RP_CLIENT_SECRET = os.environ.get("OKTA_ADMIN_CLIENT_SECRET")
OIDC_VERIFY_SSL = True
LOGIN_REDIRECT_URL = f"{BASE_URL}/admin/"
OIDC_REDIRECT_URL = f"{BASE_URL}/admin/oidc/callback/"
OIDC_AUTH_REQUEST_EXTRA_PARAMS = {"redirect_uri": OIDC_REDIRECT_URL}

It is now possible to override the login template. We added a button allowing users to connect with their Okta account.

# In templates/admin/login.html
# Admins can also log in through the usual credentials flow
{% extends "admin/login.html" %}

{% block extrastyle %}
   {{ block.super }}
   <style>
       #content-main {
           float: none !important;
       }

       .okta_login_container {
           margin-top: 15px;
           display: flex;
           justify-content: center;
       }
   </style>
{% endblock %}

{% block content %}
   {{ block.super }}

   <div class="okta_login_container">
       <a href="{% url 'oidc_authentication_init' %}">Or Login With Okta</a>
   </div>
{% endblock %}

New button to log in with Okta

After adding a way to connect through Okta to Django Admin, we must save and update users. We are getting their information from Okta and adding them in Django. To do so, we override the class OIDCAuthenticationBackend from mozilla-django-oidc. We can get the information about the user's verified email and create or update it in Django, without any permissions for the moment.

class OktaAdminAuthenticationBackend(OIDCAuthenticationBackend):
   def verify_claims(self, claims: OktaClaim) -> bool:
       return (
           super().verify_claims(claims)
           and claims.get("email_verified", False)
       )

   @transaction.atomic
   def create_user(self, claims: OktaClaim) -> User:
       user: User = self.UserModel.objects.create_user(
           claims.get("email"),
           None,  # password
           first_name=claims["given_name"],
           last_name=claims["family_name"],
       )
       user.save()

       return user

   @transaction.atomic
   def update_user(self, user: User, claims: OktaClaim) -> User:
       """Update existing user with new claims, if necessary save, and return user"""
       user.first_name = claims["given_name"]
       user.last_name = claims["family_name"]
       user.save()

       return user

Administrators can log in to the application with their Okta credentials. I will show you how to add their permissions synchronizing it with Okta.

Add groups and permissions from Okta in Django Admin

After adding users to Django admin, we must grant them the right permissions to manage the customer’s website. In my project, we had two types of administrators for each team:

Admins with basics rights. They can create documents for their team’s users but can’t publish them.

We named them Admin - [Group Name] in Okta.
Controllers. They are admins who can validate and publish documents for their team’s users.

We named them Admin - [Group Name] Controller in Okta.

Example of two groups in Okta

In Django Admin, we decided to create two types of group. The first one is a group for each team with basic permission. The other is a general group called Administrator Controller with more permissions to validate and publish documents. The graph below shows the flow to attribute groups to administrators.

User and group creation flow

To handle those permissions, we get user information from the Okta response which contains the user groups. We can separate those groups according to our business rules as described above.

def handle_controller_groups(self, groups):
    groups_with_controller = set()
    for group in groups:
        # remove Controllers groups and add generic 'Administrator Controller'
        if bool(re.search("[Cc]ontroller", group)):
            groups_with_controller.add(ADMINISTRATOR_CONTROLLER)
        else:
            groups_with_controller.add(group)
    return groups_with_controller

def get_user_groups(self, claims: OktaClaim) -> QuerySet:
    claim_groups = claims.get("groups")
    assert claim_groups
    groups = [group[8:] for group in claim_groups]  # remove `Admin - ` prefix

    groups_with_controllers = self.handle_controller_groups(groups)

    return Group.objects.filter(name__in=groups_with_controllers)

Finally, we use those functions to get users permissions when we create and update them.

def verify_claims(self, claims: OktaClaim) -> bool:
    return (
        super().verify_claims(claims)
        and claims.get("email_verified", False)
        and claims.get("groups") is not None
    )

@transaction.atomic
def create_user(self, claims: OktaClaim) -> User:
    user: User = self.UserModel.objects.create_user(
        claims.get("email"),
        None,  # password
        first_name=claims["given_name"],
        last_name=claims["family_name"],
    )
    user.groups.set(self.get_user_groups(claims))
    user.save()
    return user

@transaction.atomic
def update_user(self, user: User, claims: OktaClaim) -> User:
    """Update existing user with new claims, if necessary save, and return user"""
    user.first_name = claims["given_name"]
    user.last_name = claims["family_name"]
    user.groups.set(self.get_user_groups(claims))
    user.save()

    return user

Our administrators can create their documents with the right permissions and only their team members can see them. Besides, controller admins can manage their team and review documents to publish them.

Conclusion

With this implementation, administrators don’t need to call us to manage their teams and they only need their Okta credentials to do their various jobs.

One of the limitations is that if a new group is created, it needs to be added in Okta and Django to allow users to have their permissions when they log in. We decided we didn’t want to create groups when we didn’t recognize them in Django to avoid issues with permissions as administrators can upload sensitive information.

I hope you enjoyed reading this article and you learned a new way to handle administrators permissions with Okta and Django admin. Hope you’ll give it a try!

You can also contact one of our python expert: