Theodo

Spice up Your Website with a DocuSign Electronic Signature Embedded Tool

Fri, 14 May 2021 00:00:00 GMT

Introduction

Integrating electronic signing in a website is not always so easy. If you ever get tired of spamming users who already have an overflowing mailbox with DocuSign emails, then you have landed on the right article. Let me show you how to replace them with an elegant embedded tool that will boost your platform.

This article addresses the following problems:

Finding a simple way to make people sign a disclaimer before accessing the content of a certain page;
Using DocuSign in a modal;
Embedding everything on your website.

Why this one-page architecture?

The goal of this architecture is to make sure users have to go through the disclaimer check before accessing the page. If you've got two different pages for that purpose, any decent attacker could bypass it:

Yup, it's almost as simple as that...

Why not using DocuSign emails?

When developing a feature using DocuSign, you will probably have to pick one of these two options:

Embed DocuSign signing;
Send DocuSign emails (which seems to induce less coding thus sounds more attractive).

After going through a lot of pain playing around and debugging unsent emails, I figured that not having to worry about unsent, unread, or spam-categorized emails was a fair point for the first option. The second reason is simple: it's easier to track issues, as everything - or almost - is embedded within your website. Last but not least, it came up that having a linear path would boost the UX by removing external dependencies.

Not paramount but worth mentioning, it also makes it easier to perform E2E tests on this feature.

In overall, this solution was:

Simple enough to be quickly implemented;
Customizable enough to fit all my different cases;

Installation

Simply run pip install docusign-esign to get started.

Let's break it down

We will assume in this article that the user has not signed the disclaimer yet.

Server-side

As shown in the schema above, the backend falls in two steps :

Defining the DocuSign envelope

The backend receives user information (email, name, ...) from the frontend. Then, we use the DocuSign Python Client to define our DocuSign envelope, i.e which document(s) will be part of the envelope, where they should be signed, who should sign them, and so on. This results in an envelope definition that we send to DocuSign to create it.

Getting the signing URL

DocuSign returns an envelope ID corresponding to the created envelope. It is used to start the signing ceremony, meaning that it is sent to DocuSign to generate a signing URL. Ultimately, this signing URL is sent back to the frontend.

Bear in mind that we can pick at this stage the redirection URL, i.e to which URL the user will be redirected once the signing ceremony is finished (necessary to understand our trick!).

Macro implementation (view)

Following this logic, the implemented view should create a DocuSign envelope and return its signing url to whomever calls the related endpoint.

docusign_service = DocuSignClient()

class DocuSignViewSet(APIView):
    def get(self):
        envelope_id = docusign_service.call_docusign_create_envelope(
            full_name=full_name,
            email=email,
            contact_crm_id=contact_id,
            document_path=document_path,
            document_name=document_name,
        )
        signing_url = docusign_service.start_signing_ceremony(
            envelope_id=envelope_id,
            full_name=full_name,
            email=email,
            contact_crm_id=contact_id,
            redirection_route=<CUSTOM_REDIRECTION_ROUTE>,
        )

        return Response(signing_url)

Micro implementation (DocuSignClient class)

Click here to discover details about its implementation

To make it easier, let's create a DocuSign class with all necessary methods. This will let us keep the same valid token during the whole process and renew it if needed.

import jwt
from docusign_esign import *

class DocuSignClient:
     token = None

Now, to communicate with the DocuSign API, what we first need is a way to get a valid EnvelopesApi object, i.e that has a valid token and correct parameters (pathname and headers), so let's add these two methods:

     def get_client(self):
         api_client = ApiClient()
         api_client.host = <MY_DOCUSIGN_API_BASE_PATH>
         api_client.set_default_header("Authorization", f"Bearer {self.token}")

         return api_client

     def get_envelope_api(self):
         api_client = self.get_client()
         envelope_api = EnvelopesApi(api_client)

         return envelope_api

However, because we want the token to be valid and to automatically renew if not, I added the following pre-check:

     def get_client(self):
         if not self.is_token_valid():
             self.token = self.renew_token()

        # ...

As you will see below, this check is essentially a call to the DocuSign API using the current token to verify whether it is valid. If it is expired or invalid (401 status code), we renew it. We use a single login for all users in the platform. Therefore, we use the JWT Grant authentication as suggested in the documentation.

     def is_token_valid(self, method="GET"):
         headers = {"Authorization": f"Bearer {self.token}"}
         response = requests.request(
             method, f"{<MY_DOCUSIGN_API_BASE_PATH>}/v2.1/accounts/", headers=headers
         )

         if response.status_code != 401:
             return True

         return False

     def renew_token(self):
         current_datetime = datetime.datetime.now()
         jwt_token = jwt.encode(
             {
                 "iss": <MY_DOCUSIGN_INTEGRATION_KEY>,
                 "sub": <MY_DOCUSIGN_API_USERNAME>,
                 "name": <NAME>,
                 "aud": "account-d.docusign.com",
                 "iat": current_datetime,
                 "exp": current_datetime + datetime.timedelta(minutes=30), # validity duration of 30 minutes
                 "scope": "signature impersonation",
             },
             key=<MY_DOCUSIGN_PRIVATE_KEY>.encode(),
             algorithm="RS256",
         )

         headers = {"Content-Type": "application/x-www-form-urlencoded"}
         url = f"{<MY_DOCUSIGN_BASE_URL>}/oauth/token"
         payload = {
             "grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
             "assertion": jwt_token,
         }
         response = requests.request("POST", url, headers=headers, data=payload)

        return response.json()["access_token"]

Believe me, all these methods will simplify your life. You'll only have to call get_envelope_api() to get a functional EnvelopeApi with a valid token.

Once done, we can then add methods to create a DocuSign envelope, which is roughly a container for documents that you send to a recipient for signing.

     def make_docusign_envelope(
         self,
         signer_full_name,
         signer_email,
         contact_id,
         document_path,
         document_name,
         anchor_y_offset="0",
         anchor_x_offset="0",
     ):
         with open(document_path, "rb") as file:
             content_bytes = file.read()
             base64_file_content = base64.b64encode(content_bytes).decode("ascii")

         document = Document(
             document_base64=base64_file_content,
             name=document_name,
             file_extension="pdf",
             document_id=1,
         )

         signer = Signer(
             email=signer_email,
             name=signer_full_name,
             recipient_id="1",
             routing_order="1",
             client_user_id=contact_id,
         )

         sign_here = SignHere(
             anchor_string="/sn1/", # the anchor string to indicate where to sign on the PDF
             anchor_units="pixels",
             anchor_y_offset=anchor_y_offset,
             anchor_x_offset=anchor_x_offset,
         )

         signer.tabs = Tabs(sign_here_tabs=[sign_here])

         envelope_definition = EnvelopeDefinition(
             email_subject="Please sign this document",
             documents=[document],
             recipients=Recipients(signers=[signer]),
             status="sent",
         )

         return envelope_definition

The make_docusign_envelope() method allows you to design a fully customizable envelope, by defining all components such as the document content, signers, locations where a signature has to be attached... You can then use it as a template like so:

     def call_docusign_create_envelope(
         self, full_name, email, contact_crm_id, document_path, document_name,
     ):
         envelope_api = self.get_envelope_api()
         envelope_definition = self.make_docusign_envelope(
             full_name, email, contact_crm_id, document_path, document_name
         )

         results = envelope_api.create_envelope(
             <MY_DOCUSIGN_API_ACCOUNT_ID>, envelope_definition=envelope_definition,
         )

         envelope_id = results.envelope_id

         return envelope_id

Indeed, call_docusign_create_envelope will make use of the previous method to create an envelope based on this template and returns an envelope ID that can be used to start the signing process (a.k.a ceremony).

    def build_docusign_redirection_url(self, redirection_route, options):
         return <MY_FRONTEND_URL> + redirection_route + options

     def start_signing_ceremony(
         self,
         envelope_id,
         full_name,
         email,
         contact_crm_id,
         redirection_route,
         authentication_method="none",
     ):
         envelope_api = self.get_envelope_api()
         return_url = self.build_docusign_redirection_url(redirection_route, envelope_id)
         recipient_view_request = RecipientViewRequest(
             authentication_method=authentication_method,
             client_user_id=contact_crm_id,
             recipient_id="1",
             return_url=return_url,
             user_name=full_name,
             email=email,
         )

         results = envelope_api.create_recipient_view(
             <MY_DOCUSIGN_API_ACCOUNT_ID>,
             envelope_id,
             recipient_view_request=recipient_view_request,
         )

         return results.url

Note that the custom redirection url built by build_docusign_redirection_url is the one that will be used frontend-wise once the signing ceremony is completed, i.e when the user clicks on the "Finish" button. Completing the ceremony will also update the envelope status in DocuSign to completed.

Client-side

Add DocuSign in an iframe

When the user accesses the restricted page, the backend is called to get a signing URL. Our modal contains an iframe that will use this URL to embed DocuSign content, meaning DocuSign envelope signing portal with all documents which have to be checked.

export const DocuSignModal: React.FC = () => {
  const { signingUrl } = getDocuSignSigningUrl(); // calls the previously created DocuSignViewSet

  return (
    <Modal>
      <iframe src={signingUrl} />
    </Modal>
  );
};

Use the redirection link to update the parent component

As seen previously, once the user signed all necessary documents and clicked on DocuSign "Finish" button, the URL provided when the signing ceremony started is used to redirect the user. The issue here is that the redirection will occur within the iframe since it is a different browsing context. And of course, we do not want that.

And now the trick!

Remember what we said about having free hands on the redirection URL? Well, we used this feature to build a URL such that it points to a page containing a shadow component and contains the corresponding envelope id. It will look like so: https://my-website.com/my-redirection-page/[ENVELOPE_ID]. Once rendered, this shadow component will post a message to the parent component (i.e the modal). This message contains the envelopeId of the created envelope, which can be found in the URL.

export const MyShadowComponent: React.FC = () => {
  const router = useRouter();

  useEffect(() => {
    window.top.postMessage(
      JSON.stringify({
        success: true,
        envelopeId: String(router.query.envelopeId),
      }),
      window.location.origin
    );
  });

  return null;
};

This message sent from the iframe will be listened to by the parent component (the page itself), which can then perform actions such as closing the modal, refreshing, routing to another page...

// event.origin and window.location.origin have to be compared for security reason
const isSameOrigin = (event: MessageEvent) =>
  event.origin === window.location.origin;

useEffect(() => {
  const handler = (event: MessageEvent) => {
    if (event.data && isSameOrigin(event)) {
      const { success, envelopeId } = JSON.parse(event.data);
      if (success) {
        // Do something such as closing the modal, saving the envelopeId, ...
      }
    }
  };
  window.addEventListener("message", handler);

  return () => window.removeEventListener("message", handler);
}, []);

You're done!

Conclusion

I hope you found interest in this article which highlights a simple way to embed DocuSign in your website. There are still many ways to improve and customize this flow, with additional features such as:

Saving in the backend users who have already signed so that they are not asked to sign again later ;
Tracking documents versions.

Hope that was useful, cheers!

Useful links

How to develop on Shopify in 2021

Wed, 28 Apr 2021 00:00:00 GMT

As a software developer, I often hear that No-Code is the future of app development. That in a few years, automatically generated code will replace all the frameworks that we currently have. With my young experience in this field I'm quite sceptical, because I still believe that to truly answer a problem or a need you will always need something that can embrace it completely.

However, I was quite amazed by my last and first experience using one of the major tools in the No-Code business: Shopify.

Shopify became in the last few years the reference for online stores, and is now powering more than 18% of all eCommerce stores (according to BuiltWith). Their success can mostly be explained by the convenience of their product: by only using the platform, you can build a highly customized store, ready to use, in only 2 hours.

However, if you have very specific needs regarding your online store, you will quickly need a developer to be able to deep dive into Shopify's technical documentation. Therefore, I decided to write this article to summarize the different things that I learnt using Shopify as a developer, and how I used them in my project.

Here are the different topics that will be covered:

Front framework: Liquid
Embed a React App inside Liquid's engine
Shopify's API - Usage and Limitation
Development Workflow: Automatize your deployment with CircleCI, Github and ThemeKit

Front framework: Liquid

Even though Shopify has the specificity to be used as a headless ecommerce solution (using their API), the native behavior of Shopify is to use their embedded front system (Wordpress-like).

And for our project, we decided to stick with the classic front system as it allowed us to have an instant-ready website, giving the possibility to our client to test different configurations.

So, we had to deal with their own programming language: Liquid.

Liquid is an open-source template language written in Ruby. As many template languages (like Jinja (Python) or Twig (PHP)), it makes a bridge between data and HTML, in which you can add logic (if, while loops...). In Shopify's usage of Liquid, the data is the different entities related to your store (products, cart, collections...).

Before launching yourself into any Shopify development, I highly recommend you to read this introduction to the Liquid language, written by Keir Whitaker, a former Shopify Partner. It describes in an exhaustive way how Liquid is built and how Shopify use it for their front framework.

It also mentions one of the main downsides that we encountered during our project: Liquid doesn't have any concept of state, which can really be a pain point if you are willing to make your website a little bit interactive, by creating a real user experience. In our case, we wanted to create an interactive quiz, in which our users could answer a few questions about themselves, and push to them the best matching products.

In order to develop our feature inside Shopify's front engine, we decided to use React's framework, embedded inside a liquid template.

Embed a React App inside Liquid's engine

Our challenge here was to be able to use the ease of front development with React without creating a separated hosted webapp. Our idea was to develop the React App in the same repository, and then build the static files (as for production) and add them to the assets of a Shopify theme.

Here was our deployment script:

#!/bin/bash

APP_BUILD_FILE=build/index.html
THEME_FILE=theme_live/sections/quiz.liquid

echo 'Adapting html file for shopify asset urls'
sed -E 's/(<script src=")\/static\/js\/([a-zA-Z0-9._-]*)("><\/script>)/\1{{'"'"'\2'"'"' | asset_url }}\3/g' $APP_BUILD_FILE > $THEME_FILE

echo 'Replacing built javascript files in theme'
rm -f theme_live/assets/quiz*
cp build/static/js/* theme_live/assets/

Once a React App is built, the entry file is the index.html that contains all the script tags pointing to the js files of your app, divided in different chunks. The first command of our script catches all the script tags contained in the index.html file and then replaces them by a liquid syntax.

From:

<div id="root"></div>
<script src="/static/js/quiz_runtime-main.8923d629.js"></script>
<script src="/static/js/quiz_2.20b50bf0.chunk.js"></script>
<script src="/static/js/quiz_main.f9693573.chunk.js"></script>

We now have:

<div id="root"></div>
<script src="{{'quiz_runtime-main.8923d629.js' | asset_url }}"></script>
<script src="{{'quiz_2.b689c532.chunk.js' | asset_url }}"></script>
<script src="{{'quiz_main.d6725a08.chunk.js' | asset_url }}"></script>

The second command then removes the previous react JS files and replaces them with the new assets, in our quiz section.

However, this creates a separate app that can't interact with the liquid framework. Therefore, we needed to analyze how we could retrieve data from our Shopify store and display it to the user. Hopefully, Shopify has a great API, which allows you to interact with a lot of different elements of your stores.

Shopify's API - Usage and Limitation

This is where Shopify becomes very interesting. As said before, using the built-in Liquid framework allows you to create a ready-to-use store in only 2 hours. However, you'll sometimes find yourself in need to interact with your store data outside of your storefront, or maybe you want to only use Shopify as a backend, and make a completely custom frontend ,using gatsby or NextJS.

This gives you the possibility to move from a Liquid frontend to a much more customizable interface, by using Shopify's back API.

Shopify has 2 APIs available:

The Admin API (GraphQL and REST)
The Storefront API (GraphQL only)

Shopify's Admin API

The Admin API is the main API that allows you to interact with ALL the elements of your store. You can use it to follow your orders, products, payments... Well, pretty much everything. This API is mainly used by shopify's app developers, to extend the behavior of the basic Shopify back office.

In my case, I used it to create our product recommendation algorithm. As said before, our challenge was to create a quiz in order to advise our customers on a very specific market. We wanted to know:

The characteristics of the products they were looking for
Their family composition
What type of products they were looking for

Once the answers were given, our app then called the Admin API, in order to retrieve the best matching products. However, this is where I discovered the API's rate limitations. Indeed, our main goal was to consider all the selected product categories by our user (that could go up to 15 categories - the aim of our quiz was to create big buckets of products), and to propose in each category the best product.

We went for the GraphQL API because it allowed us to:

Make a single call with different product categories
Get only the data that we needed

However, GraphQL's API usage is limited by the cost of your query. This means that you cannot make a huge query in order to retrieve all your data at once. With classic Shopify, your query can have a maximum cost of 1000.

As we didn't have a lot of products per category, we limited the number of products per category to retrieve per 20, and limited to the maximum the data that we wanted to retrieve:

productType1: products(first: 20, query: "product_type:'productType1'") {
    edges {
        node {
            id
            productType
            tags
            priceRangeV2 {
                minVariantPrice {
                    amount
                }
            }
        }
    }
}
productType2: products(first: 20, query: "product_type:'productType2'") {
    edges {
        node {
            id
            productType
            tags
            priceRangeV2 {
                minVariantPrice {
                    amount
                }
            }
        }
    }
}

Once we retrieved the best matching products, we then sent back only the selected productIds, to give the possibility to our front to call the storefront API.

Shopify's Storefront API

Storefront's API is a way to expose your store's data to external apps (public or private), unauthenticated. The perfect usage of this API is when you want to use Shopify only as a merchant platform, and create your custom front platform (Shopify as a headless ecommerce solution).

It is important to remember that it's an unauthenticated API, meaning that once the Storefront's API is enabled, everybody can access the data that you exposed. However, you can control the data that you expose (you can only display the product details for example).

We used the storefront API for our store directly in our React App, in order to display the different products, and call the checkout to add the different items to the user's cart. We got inspired from the different adaptation of the buy-sdk adapters that you can find on Shopify's Github.

Development Workflow: Automatize your deployment with CircleCI, Github and ThemeKit

Last but not least, I will present the development workflow that we set up on our project.

Shopify has developed a command-line tool called Theme Kit, enabling you to interact with your shopify store and deploy code instantly.

Once Theme Kit is installed on your computer/server, you need to configure it inside your app folder, following this tutorial.

Theme Kit allows you to (in particular):

Preview your code as you code it on your shopify store theme using the theme watch command
Deploy code to one of your shop theme using theme deploy --env=development or any other name that you would have defined in your config.yml file.

development:
  password: ${SHOPIFY_PASSWORD}
  theme_id: ${SHOPIFY_DEV_THEME_ID}
  store: store-name.myshopify.com
  timeout: 1m0s
production:
  password: ${SHOPIFY_PASSWORD}
  theme_id: ${SHOPIFY_THEME_ID}
  store: store-name.myshopify.com
  timeout: 1m0s

Each developer had their own development theme (you can have up to 20 themes on one shopify store), and their was one "live theme" which was the production theme. The different useful variables were stored in a variables file, that Shopify's engine can interpolate. For further information, you can check Theme Kit configuration documentation.

Here was our developer workflow:

Get the latest code from the master branch using git
Push my code to my Shopify development theme using theme deploy --env=development
Make modification while using theme watch (so that you could see directly the changes on your Shopify theme)
Push your code, review it on github
Merge on master.
Once the code was merged, we had a CI/CD script that:
- Built the React App and moved the built files into the assets of our shopify theme
- Deploy the code to Shopify using Theme Kit

The last step was done using the following script in your CI:

echo 'Deploying runinng cd theme_live && theme deploy --allow-live'
cd theme_live && theme deploy --allow-live --env=production

The Shopify production Theme ID and user credentials were injected inside our CI configuration, to be able to deploy to the production environnement.

Conclusion

As you can see, on top of being a very easy-to-use product for non-tech people, Shopify is also a real state-of-art product. They thought of a simple workflow to automate your deployment, you can easily access your store using a highly documented API.

They created a real game changer when it comes to build an ecommerce website: you can start with a prototype using the classic Liquid templating product, then if you want to fully customize your storefront, you can change to a headless CMS version of Shopify using their API, and then you can even scale up by using their premium version Shopify Plus, allowing to manage different stores in different country.

And one thing that is important to notice: once your store is created, it's already usable. Shopify has their own secured payment platform, PIM management, and many other features that are required in every ecommerce store, which you don't need to develop anymore. And icing on the cake, by using Liquid's framework, your SEO will be optimized and fully manageable from your store.

I highly recommend each developer to take the time (with Shopify's 14 days trial) to build their own Shopify store, as it's a great skill to have in a world where the shopping experience is becoming more and more digitized.

How to Check Your Website on Mobile in the Local Environment

Wed, 28 Apr 2021 00:00:00 GMT

At Theodo, we develop solutions for both mobile and desktop. Most of the time we use the chrome device toolbar to see the visual rendering of the website when we develop in the local environment. Then when the code is deployed, we also check in the development/staging environment that everything works correctly.

However, sometimes, this is not enough. For example, this is not applicable to debug a behavior that occurs only on mobile devices. The obvious solution: finding a way to check the website on mobile, in the local environment. There are several ways to do that and each has its perks and drawbacks. In this article, I will present you with those solutions to help you find the one that suits you best.

For you to find the best solution in your use case, I will evaluate each solution with the following criteria:

📱 ‍Provides an experience similar to the end-user interaction on mobile
📋 Easy access to logs
🔧 Easy to set up

Chrome device toolbar

Before you start, make sure that the feature you want to check cannot be checked with the Chrome device toolbar.

Chrome device toolbar is accessible by a single click on the mobile icon at the top of the Chrome inspector

It does what you expect and maybe more…

Chrome device toolbar is an obvious solution, known by most developers, that allows you to test responsiveness on a large variety of devices. But responsiveness is not the only thing you can test with the chrome device toolbar. All the available options are presented in great detail in the chrome documentation. If you need an incentive to go see for yourself, be aware that the device toolbar allows you to override geolocation, throttle your CPU and/or internet speed, display media queries… A less known feature of the Chrome device toolbar is that it properly sets the user-agent according to the device you choose. So if you implement a different behavior based on the userAgent, you will be able to see it.

But Chrome device toolbar is not an emulator

However, as complete as it is, the chrome device toolbar is still limited. Here is a quick list of things you may need, and that is still unavailable with this tool:

The fluidity of scrolling and swiping. One of the main differences between desktop and mobile is the way the user interacts with the device. And this is a big drawback of the Chrome device toolbar. Your scroll could be lagging like hell on a mobile device, you would not be able to see it.
Opening/closing of keyboard and focus. Don't forget that the Chrome device toolbar remains more or less a desktop website visualizer that can take the shape of a mobile device. It won't display the behaviors that are particular to mobile such as the opening of a keyboard.
Running code that is specific to an operating system. A video player designed to run on IOS won't run in the chrome toolbar even if you selected an iPhone.

Solution evaluation

📱 Close to end-user experience: Mediocre

❌ Chrome device toolbar is not an emulator.

📋 Access to logs: Excellent

✅ No need to change development flow to retrieve logs. Everything you need is directly available in the browser.

🔧 Easy to set up: Excellent

✅ A click on a button in the inspector.

Chrome device toolbar is enough for basic responsiveness but what if we need to access our website / debug on a real mobile device ?

Using your local network to access your website in the local environment

There is a quick solution that doesn't require any third-party software and that allows you to connect to your website from your mobile on your local network. All you need is a mobile device and a computer connected to the same network, i.e. a wifi network, or a shared mobile internet connection.

The principles

As you probably already know, when setting up the services for your application, you give them an address which on your local environment is localhost + port number. You can make your website accessible to all the users on the local network by exposing it to your address on this network.

The reason for this is rooted in the difference between localhost and your local IP address. Localhost is an internal address in your computer used by the loopback interface, a virtualized network interface. Localhost is only used inside your computer. On the other end, the local IP address is an address given by your router for your computer to communicate with other devices on the local network.

How to set it up

Identify your local Ip address. On Windows, you can open the terminal and execute ipconfig. On Mac and Linux, the command to execute is ifconfig. But maybe you won't even need to do that. For example, when starting a React App, the local IP address is displayed in the console.

You can see your local IP address when launching a React App as "On Your Network"

If you only have a frontend, you are already good to go. Just enter the local IP address in the URL bar of your browser with the right port, and you will see your website.
If you have a backend, you should replace your backend adress(es) called by the frontend from

http://localhost:<your_backend_port>

http://<your_local_ip_address>:<your_backend_port>

Your website is now accessible at the following address on both your desktop device and to any device connected to the local network:

http://<your_local_ip_address>:<your_front_port>

A nice addition to this is to add a script such as the one below in your package.json to automatically fetch your local IP address and update your frontend/backend URLs in your environment files:

#grepping the local ip address
LOCAL_IP=$(ifconfig
| grep -Eo 'inet (addr:)?([0-9]*\\.){3}[0-9]*'
| grep -Eo '([0-9]*\\.){3}[0-9]*'
| grep -v '127.0.0.1');
# using a .env.mobile file as a template
# and moving the output into .env file
sed \"s,frontend_url,$LOCAL_IP,g\" .env.mobile > .env;
# starting the react app with the updated file
react_app start

Make CORS allowed host changes if needed. You will only have to do the same replacement as the previous point for your front-end address.

How to see the logs

Now that you are accessing the website in your local environment on a mobile device, the next step would be to see the logs of your device so you can develop and debug your application correctly.

MacOS ⇔ IOS. For this, you can use the Safari development tools. Connect your device to your computer using a USB cable and when asked if you trust the computer, answer yes. Open Safari and click on the 'Develop' tab

You can access your mobile logs by opening the Safari Browser. Select first the 'Develop' tab then the 'name of your device' and finally the 'name of the page'
Mac/Windows/Linux ⇔ Android: Google is providing a tool to access logs by connecting your mobile to your computer using a USB cable .
Windows/Linux ⇔ IOS: It seems there is no obvious solution for this currently.

A versatile solution that covers (almost) any use case

What is great about this solution is that it allows you to test directly on your mobile device and also that it works not only for a frontend but also for a "complete website" with a/multiple backend and a database as well.

But:

This solution requires updating all the addresses throughout your application. However, this can be easily solved by adding a script to your package.json
Some third-party API requires HTTPS and won't work with this setup.
Your router may change your local IP address from time to time.

Solution evaluation

📱 Close to end-user experience: Good

✅ You are debugging directly on your mobile device.
❌ You can only use this method on the devices you own, so you may overlook some edge cases on other devices.
❌ You also may encounter errors if HTTPS is mandatory for some APIs you use. In that case, see my next solution.

📋 Access to log: Very good - No Logs

✅ The same quality of logs as when you are developing for computer users...
❌ ...Except for the IOS + Windows/Linux duo.

🔧 Easy to set up: Very good

✅ Relatively easy to put in place and to automatize
✅ You don't need to install anything.
❌ A bit time-consuming to set up the first time

Now we can access our website on a real mobile device. But what if we need HTTPS ?

Using LocalTunnel to access your website in the local environment from your mobile with HTTPS

What is LocalTunnel?

LocalTunnel is a service that allows you to expose an endpoint on both a public HTTP and a public HTTPS address.

If you don't know localTunnel, maybe you are more familiar with ngrok its best-known counterpart. However, the main advantage of localTunnel over ngrok is that you don't have a limit on the number of endpoints you can create whereas you have to pay as soon as the second one for ngrok.

How to set it up?

You have two options to use LocalTunnel on your project

Start the tunnels using the command line using: lt --port <port>

Running localTunnel using the command line is pretty straightforward
Using the npm package directly in a node project

const localtunnel = require("localtunnel");
const myTunnel = await localtunnel({ port: 3000 });
console.log(myTunnel.url);

You will still have to update your URLs in your configuration file (same principles as for the local IP address). I personally prefer the second solution because the address returned by LocalTunnel is random and the task of updating the URL for your backend/frontend in your configuration files can quickly become annoying. I developed a quick example of an independent node project that sets up your configuration files when launching the app.

Concerning logs, they are still available using the same principles as for the local IP address.

The ultimate solution to local development on mobile?

Pros:

You are making your website accessible to anyone on the internet and not only on your local network anymore. You want your client/another developer/your manager to see the new feature before deploying it, this is now possible!
LocalTunnel provides an HTTPS address so you won't have any issue related to HTTP anymore.

Cons

Complex to set up. Each time you launch localTunnel, the addresses provided are different which makes it a bit tricky to set up with a script even though it remains possible.
You can only test on the devices/browser versions you have access to.
Even if there is little chance of anything happening you are still making your website public. Make sure you don't display any critical information.

Solution evaluation

📱 Close to end-user experience: Very Good

✅ Directly on your mobile device
✅ In HTTPS
❌ Only the mobile you own / the browser version you have access to

📋 Access to log: Very good - No Logs.

✅ The same quality of logs as when you are developing for computer users...
❌ ...Except for the IOS + Windows/Linux duo.

🔧 Easy to set up: So-so.

✅ Can be automated
❌ More time-consuming than the local-network solution

Now you can access to your website on a mobile device in HTTPS. But you may wonder what to do when you don't have the device / the browser version you want to check

Emulators/simulators

You may be tempted to use an emulator / a simulator to access your local website. Indeed, using Android Studios or Xcode Ios Simulator to access your website during local development is possible but here are three reasons why this is probably a bad idea:

Emulators are heavy. Your computer will soon be over-encumbered with mobile images
Emulators are slow. You have to imitate the behavior of smartphone components on your computer and it requires a lot of calculation power making your application slow/laggy in the emulator.
Emulators are not designed for web development. Accessing quality logs can be really hard.

If you want to test a specific mobile or a specific browser version, your best bet is probably to use BrowserStack.

Our decision tree is complete !

Takeaway

Two last things that can help you find the best solution for your project :

Don't go for the most technical solution if all you need is to test basic responsiveness. The chrome device tool is already very complete.
If you want to test complex behavior, you will probably know it beforehand. So set up your project for mobile testing at the beginning. You will lose far less time on the way.

How To Make Tree Shakeable Libraries

Tue, 13 Apr 2021 00:00:00 GMT

At Theodo, our aim is to build reliable and fast applications for our customers. Some of our projects include improving the performance of already existing applications. During one of these missions, we managed to reduce the bundle size of all our pages by a whopping 500KB Gzipped by tree shaking our internal libraries.

While doing so, we realized tree shaking is not something we can simply turn on and off. There are many factors that impact the tree shaking quality of libraries.

This article's aim is to provide an exhaustive guide on making an optimized tree shakeable library. If I had to summarize the procedure:

Check whether the library is tree shakeable by testing it against a known application in a controlled environment
Use ES6 modules so bundlers can detect unused export statements
Use the side effects optimization and make your library side effects free
Split the library logic in multiple small modules and preserve the library's module tree
Do not lose the module tree or the ES modules characteristics when transpiling your library
Use the latest version of a tree shaking capable bundler

What is tree shaking and why is it important?

From the MDN docs:

Tree shaking is a term commonly used within a JavaScript context to describe the removal of dead code.

It relies on the import and export statements in ES2015 to detect if code modules are exported and imported for use between JavaScript files.

Tree shaking is a way to achieve dead code elimination by detecting which exports are unused in our application. It is performed by application bundlers such as Webpack or Rollup but was initially implemented by Rollup.

So why call it tree shaking? One can visualize the application's exports and imports in the form of a tree. The healthy leaves and branches represent used imports of the application while dead leaves symbolize unused code separated from the rest of the tree. Shaking the tree would eliminate all our unused code.

Why is this important? Tree shaking can have a huge impact on your browser applications. The more code gets bundled in the app, the more time the browser will spend downloading, decompressing, parsing, and executing it. Removing unused code is thus of the utmost importance to make the fastest applications possible.

There are many articles and resources out there explaining tree shaking and dead code elimination. Here we will be focusing on libraries that are consumed by applications. A library is considered to be tree shakeable if a consumer application can successfully eliminate the unused parts of the library.

But before we try to make a library tree shakeable, let's first see how we can distinguish one.

Realize a library is not tree shakeable in a controlled environment

This may sound obvious at first glance. But I noticed a lot of developers assume their library is tree shakeable because it uses ES6 modules (more on that later) or because they have a tree shaking friendly configuration. Unfortunately, this does not automatically imply that your library is in fact tree shakeable!

This brings us to the relevant question: how can one efficiently check that a library is tree shakeable?

To do so, we need to understand two things:

It is the app's bundler that will ultimately tree shake the library's code, not the library's (if it even has a bundler at all!). After all, only the app knows which parts of our library are used.
The library's job is to make sure it can be tree shaken by the final bundler

To check whether our library is tree shakeable we will be testing it against a reference application in a controlled environment:

Create a simple application (reference app) with a bundler you know how to configure and that supports tree shaking (eg Webpack or Rollup)
Set the library you want to test as a dependency of the created application
Import only one element of the library and check the output of the application's bundler
Check that the bundled output only contains the imported element and its dependencies

This strategy will make the test independent of our existing applications. It makes it easier but also allows us to play with the library without breaking anything. It also makes sure the issue does not come from the application bundler configuration.

We will be doing this for our library called user-library that we will test against a user-app application bundled with Webpack. You may use whatever bundler you feel more comfortable with.

The user-library's code looks like this:

export const getUserName = () => "John Doe";

export const getUserPhoneNumber = () => "***********";

It just exports 2 functions in an index file that we can use via an NPM package.

Let's make our simple user-app:

package.json

{
  "name": "user-app",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "build": "webpack"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "webpack": "^5.18.0",
    "webpack-cli": "^4.3.1"
  },
  "dependencies": {
    "user-library": "1.0.0"
  }
}

Notice we are using user-library as a dependency.

webpack.config.js

const path = require("path");

module.exports = {
  entry: "./src/index.js",
  output: {
    filename: "main.js",
    path: path.resolve(__dirname, "dist"),
  },
  mode: "development",
  optimization: {
    usedExports: true,
    innerGraph: true,
    sideEffects: true,
  },
  devtool: false,
};

To understand the Webpack configuration of our reference application, we need to understand how Webpack does tree shaking. Tree shaking is performed using the following steps:

Identify the application entry file (determined in the Webpack configuration)
Create an application module tree by looping through all the dependencies and sub dependencies imported by the entry file
Identify for each module in the tree which export statements are not imported by the other modules
Eliminate the unused exports and their related code using minification tools like UglifyJS or Terser

This process in done only in production mode.

The issue with production mode is minification. It becomes very hard to see whether our tree shaking worked because we cannot actually see our named functions in our bundled code.

To get around this, we run Webpack in development mode while still identifying which code is unused and would be removed in production with the optimization property set to:

  optimization: {
    usedExports: true,
    sideEffects: true,
    innerGraph: true,
  }

The usedExports property allows Webpack to identify which module exports are not used by other modules. The other two will be discussed later in the article. For now, let's just say they improve the tree shaking quality of our application.

Our user-app entry file: src/index.js

import { getUserName } from "user-library";

console.log(getUserName());

Once bundled, we now analyze the output:

/***/ "./node_modules/user-library/dist/index.js":
/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((__unused_webpack_module, exports) => {

var __webpack_unused_export__;

__webpack_unused_export__ = ({ value: true });

const getUserName = () => 'John Doe';

const getUserPhoneNumber = () => '***********';

exports.getUserName = getUserName;
__webpack_unused_export__ = getUserPhoneNumber;
/***/ })

Webpack regroups all our code in a single file. Looking at the getUserPhoneNumber export, we notice that Webpack has marked it as unused. It will be removed in production mode while getUserName is exported as it is used by our index.js entry file.

The library is tree shaken! You may repeat this step but for multiple imports and look at the output code. The objective is to make sure unused code in the library is marked as unused by Webpack.

Everything looks fine for our very simple user-library. Let's make it a bit more complicated and as we do so, we will look at some tree shaking requirements and optimizations.

Use ES6 modules so bundlers can detect unused exports

Ok, this requirement is classic and well documented but is in my opinion a bit misleading. I oftentimes hear developers say we need to use ES6 modules so that our library can be tree shaken. While this statement is completely true, there is this misconception that using ES6 modules is enough to just make tree shaking work. Well, were it that simple, you definitely would not have gotten this far in the article!

However, using ES6 modules still is a requirement for tree shaking.

There are a lot of formats in which JavaScript can be bundled: ESM, CJS, UMD, IIFE and so on.

To make things simple, we will consider only two: Ecma Script Modules (ESM or ES6 modules) and CommonJS modules (CJS) as they are the most widely used for application libraries. Most libraries will use CJS modules because it allows them to be run in a node application (though Node now supports ESM). ES modules appeared way later than CJS in 2015 with ECMAScript 2015 (also known as ES6) and is considered to be the standardized module system for JavaScript.

CJS syntax example

const { userAccount } = require("./userAccount");

const getUserAccount = () => {
  return userAccount;
};

module.exports = { getUserAccount };

ESM syntax example

import { userAccount } from "./userAccount";

export const getUserAccount = () => {
  return userAccount;
};

The big difference between both is that ESM imports are static whereas CJS imports are dynamic meaning we could do the following with CJS but not with ESM:

if (someCondition) {
  const { userAccount } = require("./userAccount");
}

While this seems to be more flexible, it also means bundlers cannot make a valid application tree at compile or bundle time. The someCondition variable will be known only at runtime forcing the bundler to import userAccount in any case at compile time. This leads bundlers to just include all the CJS style imports directly in the bundle without being able to check whether the imports are actually used.

Let's modify our user-library to show this. And to make it a little bit more realistic, it will now have two files:

src/userAccount.js

const userAccount = {
  name: "user account",
};

module.exports = { userAccount };

src/index.js

const { userAccount } = require("./userAccount");

const getUserName = () => "John Doe";

const getUserPhoneNumber = () => "***********";

const getUserAccount = () => userAccount;

module.exports = {
  getUserName,
  getUserPhoneNumber,
  getUserAccount,
};

We keep the same entry file in our user-app so we do not use the getUserAccount function nor its dependency.

/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((module, __unused_webpack_exports, __webpack_require__) => {

const { userAccount } = __webpack_require__(/*! ./userAccount */ "./node_modules/user-library/dist/userAccount.js")

const getUserName = () => 'John Doe'

const getUserPhoneNumber = () => '***********'

const getUserAccount = () => userAccount

module.exports = {
	getUserName,
	getUserPhoneNumber,
	getUserAccount
}
/***/ }),

/***/ "./node_modules/user-library/dist/userAccount.js":
/*!*******************************************************!*\
  !*** ./node_modules/user-library/dist/userAccount.js ***!
  \*******************************************************/
/***/ ((module) => {

const userAccount = {
	name: 'user account'
}

module.exports = { userAccount }
/***/ })

All three exports still appear and are not marked by Webpack as unused. This is also the case for our userAccount file which will be included in the bundle.

Now let's look at the same example but with ESM by just replacing the require and exports syntax with their ESM counterparts.

/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {
/* harmony export */ __webpack_require__.d(__webpack_exports__, {
/* harmony export */   "getUserName": () => /* binding */ getUserName
/* harmony export */ });
/* unused harmony exports getUserAccount, getUserPhoneNumber */
/* harmony import */ var _userAccount_js__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(/*! ./userAccount.js */ "./node_modules/user-library/dist/userAccount.js");

const getUserName = () => 'John Doe';

const getUserPhoneNumber = () => '***********';

const getUserAccount = () => userAccount;

/***/ }),
/***/ "./node_modules/user-library/dist/userAccount.js":
/*!*******************************************************!*\
  !*** ./node_modules/user-library/dist/userAccount.js ***!
  \*******************************************************/
/***/ ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {

/* unused harmony export userAccount */
const userAccount = {
	name: 'user account'
};
/***/ })

Notice that getUserAccount and getUserPhoneNumber are marked as unused. But so does the userAccount export in the other file. Thanks to the innerGraph optimization, Webpack is able to link the userAccount import in the index file to the getUserAccount export. This allows Webpack to work recursively from the entry file and go through all of its dependencies to know which exports are unused in every module. Since Webpack knows that getUserAccount is unused, it can go and check its dependencies in the userAccount file and do the same work there etc.

ES modules allow us to look for exported code that is used or unused in our application explaining why this module system is so important for tree shaking. It also explains why one should use dependencies that export a ES module compatible build such as lodash-es, the ESM equivalent of the popular lodash library.

This being said, using only ES modules for tree shaking is still a suboptimal approach. In our example, we noticed Webpack worked recursively in each file to see whether exported code was used or unused. In this case, Webpack could have just totally ignored the userAccount file because the only import coming from that file is unused! This leads us to the notion of side effects discussed in the next part of the article.

To sum up this first part:

ESM is a requirement for tree shaking but is not enough to make it optimal
Make sure to always provide an ESM build of your library! If your consumers require both ESM and CJS builds, provide them both through the package.json with the main and module fields.
Make sure to use ESM dependencies when possible as they won't be tree shaked otherwise.

Use the side effects optimization to make your library side effects free

According to the Webpack docs, tree shaking can be split in two optimizations:

usedExports: Determine which exports of a module are used and unused
sideEffects: skip over modules that do not have any used exports and that are side effects free

To illustrate side effects, let's take the example that we used before:

import { userAccount } from "./userAccount";

function getUserAccount() {
  return userAccount;
}

If getUserAccount is unused, can the bundler assume that the userAccount module can be removed as well? The answer is no! userAccount could do all kinds of stuff affecting other parts of the app. It could inject some variables inside a globally accessible value, like the DOM for instance. It could also be a css module injecting style inside the document. But I think the best example is polyfills as we usually import them like:

import "myPolyfill";

Now this module definitely has side effects as it affects the code of the whole app as soon as it is imported. Bundlers will see this module as a possible candidate for removal as we are not using any of its exports. But removing this would break our app.

Bundlers like Webpack or Rollup will therefore see every module of our library as filled with side effects by default.

In our case, we know that our library is side effect free! We can therefore help our bundler by informing it of this. To do so, most bundlers can read the sideEffects property inside the package.json file. It is by default set to true when unspecified (there are side effects in every module of the package). You can set it to false (no side effects anywhere) or you may specify an array of files that have side effects.

We add this option to the package.json file of our library:

{
  "name": "user-library",
  "version": "1.0.0",
  "description": "",
  "sideEffects": false,
  "main": "dist/index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}

We then rerun our Webpack build:

/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ (__unused_webpack_module, __webpack_exports__, __webpack_require__) => {
  /* harmony export */ __webpack_require__.d(__webpack_exports__, {
    /* harmony export */ getUserName: () => /* binding */ getUserName,
    /* harmony export */
  });
  /* unused harmony exports getUserAccount, getUserPhoneNumber */

  const getUserName = () => "John Doe";

  const getUserPhoneNumber = () => "***********";

  const getUserAccount = () => userAccount;
  /***/
};

We see that the userAccount file has been removed from the bundle. We still see getUserAccount that references userAccount but this function has been marked by Webpack as dead code and it will be removed during minification.

The sideEffects flag is especially important for libraries that export their API through an index file that itself exports functions or variables from internal files. Without the side effects optimization, our bundlers would have to parse all the files where our exported variables are defined.

As noted by Webpack: "sideEffects is much more effective since it allows to skip whole modules/files and the complete subtree."

To better understand the difference between how both optimizations intervene:

sideEffects allows us to completely skip an imported module if none of its imported contents are used.
usedExports allows us to completely remove exports that are never imported by any module.

Ok but how is the result of skipping files different from just saying the exports of those files are unused?

Most of the time, tree shaking a library with and without the side effects optimization will give the same result. The same amount of code will be included in the final bundle. However, this is not true in some situations when analyzing the code for unused exports becomes too complex. The next part covers two of these situations where only the combination of small modules with the sideEffects optimization provides the best result.

To sum up this part:

Tree shaking consists of two parts: the used exports optimization and the side effects optimization
The side effects optimization is way more effective than just detecting unused export statements in every module
Make your library side effect free
Make sure to tell that your library is side effects free through the sideEffects property in the package.json file.

Preserve the library's module tree and split your code in small modules to fully benefit from the `sideEffects` optimization

You may have noticed that the example user-library we are using in this article is not bundled. The library just exposes a few JS files that I manually added.

Oftentimes, libraries will be bundled for multiple reasons:

Manage custom import paths
The library uses specific languages like SASS or TS that need to be transformed to CSS or JS for instance
The library needs to be available in multiple formats (ESM, CJS, IIFE ...)

Popular bundlers like Webpack, Rollup, Parcel or ESBuild are made to provide a bundle that can be served to a browser. They will thus have a tendency to create a single file regrouping all of your code so that only a single JS file needs to be sent through the network.

From a tree shaking perspective, this creates one problem: The side effects optimization is non existent as no modules can be skipped.

We are going to showcase two situations where splitted modules combined with the sideEffect optimization are essential for tree shaking.

A library module imports a CJS formatted dependency

To demonstrate the issue, we are going to bundle our library using Rollup. And we are going to have one of our library modules import a CJS formatted dependency: Lodash

rollup.config.js

export default {
  input: "src/index.js",
  output: {
    file: "dist/index.js",
    format: "esm",
  },
};

userAccount.js

import { isNil } from "lodash";

export const checkExistance = (variable) => !isNil(variable);

export const userAccount = {
  name: "user account",
};

Notice we are now exporting checkExistance and we import it in our library index file.

Here is the output in dist/index.js

import { isNil } from "lodash";

const checkExistance = (variable) => !isNil(variable);

const userAccount = {
  name: "user account",
};

const getUserAccount = () => {
  return userAccount;
};

const getUserPhoneNumber = () => "***********";

const getUserName = () => "John Doe";

export { checkExistance, getUserName, getUserPhoneNumber, getUserAccount };

Everything is bundled within a single file. Notice also that Lodash is imported at the top. We are still importing the same functions in our application meaning checkExistance is not used. However, after running Webpack, we see that the whole Lodash library is imported even though checkExistance is marked as unused:

/***/ "./node_modules/user-library/dist/index.js":
/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {

"use strict";
/* harmony export */ __webpack_require__.d(__webpack_exports__, {
/* harmony export */   "getUserName": () => (/* binding */ getUserName)
/* harmony export */ });
/* unused harmony exports checkExistance, userAccount, getUserPhoneNumber, getUserAccount */
/* harmony import */ var lodash__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(/*! lodash */ "./node_modules/user-library/node_modules/lodash/lodash.js");
/* harmony import */ var lodash__WEBPACK_IMPORTED_MODULE_0___default = /*#__PURE__*/__webpack_require__.n(lodash__WEBPACK_IMPORTED_MODULE_0__);

const checkExistance = (variable) => !isNil(variable);

const userAccount = {
  name: "user account",
};

const getUserPhoneNumber = {
	number: '***********'
};

const getUserAccount = () => {
	return userAccount
};

const getUserName = () => 'John Doe';

/***/ }),

/***/ "./node_modules/user-library/node_modules/lodash/lodash.js":
/*!*****************************************************************!*\
  !*** ./node_modules/user-library/node_modules/lodash/lodash.js ***!
  \*****************************************************************/
/***/ (function(module, exports, __webpack_require__) {

/* module decorator */ module = __webpack_require__.nmd(module);
var __WEBPACK_AMD_DEFINE_RESULT__;/**
 * @license
 * Lodash <https://lodash.com/>
 * Copyright OpenJS Foundation and other contributors <https://openjsf.org/>
 * Released under MIT license <https://lodash.com/license>
 * Based on Underscore.js 1.8.3 <http://underscorejs.org/LICENSE>
 * Copyright Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors
 */
// ...

Webpack is not able to tree shake Lodash because of its CJS format. This is a shame as we explicitly organized our library so Lodash would be imported only in the userAccount module that is unused in our application. If the module structure was preserved, Webpack would have detected that no userAccount exports were used and would simply have skipped the module and thus its Lodash import thanks to the sideEffects optimization.

In rollup, we can use the preserveModules option to preserve the module structure of our library.. Equivalents exist for other bundlers.

export default {
  input: "src/index.js",
  output: {
    dir: "dist",
    format: "esm",
    preserveModules: true,
  },
};

Rollup now keeps the original file structure. We can now run Webpack again:

/***/ "./node_modules/user-library/dist/index.js":
/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {

/* harmony export */ __webpack_require__.d(__webpack_exports__, {
/* harmony export */   "getUserName": () => (/* binding */ getUserName)
/* harmony export */ });
/* unused harmony export getUserAccount */

const getUserAccount = () => {
	return userAccount
};

const getUserName = () => 'John Doe';

/***/ })

Lodash is now skipped along with the userAccount module.

Code splitting

Preserving splitted module structure alongside the sideEffects optimization also benefits Webpack code splitting, a key performance optimization for big applications. Code splitting is widely used in web applications with multiple pages. Frameworks like Nuxt or Next both use page by page code splitting.

To illustrate the benefit, we will look at what happens when the library is bundled in a single file.

user-library/src/userAccount.js

export const userAccount = {
  name: "user account",
};

user-library/src/userPhoneNumber.js

export const userPhoneNumber = {
  number: "***********",
};

user-library/src/index.js

import { userAccount } from "./userAccount";
import { userPhoneNumber } from "./userPhoneNumber";

const getUserName = () => "John Doe";

export { userAccount, getUserName, userPhoneNumber };

In order to code split our user application, we will use the Webpack import syntax.

user-app/src/userService1.js

import { userAccount } from "user-library";

export const logUserAccount = () => {
  console.log(userAccount);
};

user-app/src/userService2.js

import { userPhoneNumber } from "user-library";

export const logUserPhoneNumber = () => {
  console.log(userPhoneNumber);
};

user-app/src/index.js

const main = async () => {
  const { logUserPhoneNumber } = await import("./userService2");
  const { logUserAccount } = await import("./userService1");

  logUserAccount();
  logUserPhoneNumber();
};

main();

The app bundle now has 3 files: main.js, src_userService1_js.main.js and src_userService2_js.main.js. Taking a closer look at src_userService2_js.main.js, we can see that the entire user-library bundle is added:

(self["webpackChunkuser_app"] = self["webpackChunkuser_app"] || []).push([
  ["src_userService1_js"],
  {
    /***/ "./node_modules/user-library/dist/index.js":
      /*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
      /***/ (
        __unused_webpack_module,
        __webpack_exports__,
        __webpack_require__
      ) => {
        "use strict";
        /* harmony export */ __webpack_require__.d(__webpack_exports__, {
          /* harmony export */ userAccount: () => /* binding */ userAccount,
          /* harmony export */ userPhoneNumber: () =>
            /* binding */ userPhoneNumber,
          /* harmony export */
        });
        /* unused harmony export getUserName */
        const userAccount = {
          name: "user account",
        };

        const userPhoneNumber = {
          number: "***********",
        };

        const getUserName = () => "John Doe";

        /***/
      },

    /***/ "./src/userService1.js":
      /*!*****************************!*\
  !*** ./src/userService1.js ***!
  \*****************************/
      /***/ (
        __unused_webpack_module,
        __webpack_exports__,
        __webpack_require__
      ) => {
        "use strict";
        __webpack_require__.r(__webpack_exports__);
        /* harmony export */ __webpack_require__.d(__webpack_exports__, {
          /* harmony export */ logUserAccount: () =>
            /* binding */ logUserAccount,
          /* harmony export */
        });
        /* harmony import */ var user_library__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(
          /*! user-library */ "./node_modules/user-library/dist/index.js"
        );

        const logUserAccount = () => {
          console.log(user_library__WEBPACK_IMPORTED_MODULE_0__.userAccount);
        };

        /***/
      },
  },
]);

userAccount is not marked as unused even though userService2 only uses userPhoneNumber... But why ?

We need to keep in mind that the usedExports optimization checks for used exports only within a module's scope. Only from there can Webpack remove unused code. From the perspective of our library module, both userAccount and userPhoneNumber are actually used. In this case, Webpack is not able to make a difference between the imports of userService1 and userService2 as seen on the following graph (both userAccount and userPhoneNumber are in green):

This means that Webpack is not able to tree shake the exports of each chunk independently when only relying on the usedExports optimization.

We now preserve our modules when bundling our library to allow for the sideEffects optimization:

Webpack still outputs the same 3 files but this time, src_userService2_js.main.js only contains the code coming from userPhoneNumber:

(self["webpackChunkuser_app"] = self["webpackChunkuser_app"] || []).push([
  ["src_userService2_js"],
  {
    /***/ "./node_modules/user-library/dist/userPhoneNumber.js":
      /*!***********************************************************!*\
  !*** ./node_modules/user-library/dist/userPhoneNumber.js ***!
  \***********************************************************/
      /***/ (
        __unused_webpack_module,
        __webpack_exports__,
        __webpack_require__
      ) => {
        "use strict";
        /* harmony export */ __webpack_require__.d(__webpack_exports__, {
          /* harmony export */ userPhoneNumber: () =>
            /* binding */ userPhoneNumber,
          /* harmony export */
        });
        const userPhoneNumber = {
          number: "***********",
        };

        /***/
      },

    /***/ "./src/userService2.js":
      /*!*****************************!*\
  !*** ./src/userService2.js ***!
  \*****************************/
      /***/ (
        __unused_webpack_module,
        __webpack_exports__,
        __webpack_require__
      ) => {
        "use strict";
        __webpack_require__.r(__webpack_exports__);
        /* harmony export */ __webpack_require__.d(__webpack_exports__, {
          /* harmony export */ logUserPhoneNumber: () =>
            /* binding */ logUserPhoneNumber,
          /* harmony export */
        });
        /* harmony import */ var user_library__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(
          /*! user-library */ "./node_modules/user-library/dist/userPhoneNumber.js"
        );

        const logUserPhoneNumber = () => {
          console.log(
            user_library__WEBPACK_IMPORTED_MODULE_0__.userPhoneNumber
          );
        };

        /***/
      },
  },
]);

src_userService1_js.main.js behaves the same way as it includes only the userAccount module from our library.

Looking at the graph, we still see that userAccount and userPhoneNumber are still considered as used exports as they are used at least once in our application. However, this time the sideEffects optimization is able to skip the userAccount module because it is never imported by userService2. The same thing happens for userPhoneNumber and userService1.

We can now understand that preserving the original module structure of the library is important. However, preserving this is useless if the original structure only has one module such as an index.js file with all the code inside. In order to make an optimal tree shakeable library, its code should be divided in multiple small modules that each handle one piece of the logic.

To use the tree analogy, one should see each leaf of the tree as a module. Smaller and weaker leafs will fall better when the tree is shaked! If the tree has fewer and bigger leafs, shaking it will not give the same result.

To sum up this part:

We should preserve the module structure of the library in order to fully benefit from the sideEffects optimization.
Libraries should be split in multiple small modules, each module exporting only one piece of logic
Tree shaking libraries in applications that use code splitting will only work with the sideEffects optimization.

Do not lose the module tree or the ES modules characteristics when transpiling your library

Bundlers are not the only tools that can harm the tree shaking of your library. Transpilers are also known to have an undesirable effect on tree shaking by removing ES modules and by not preserving the module tree.

One of the objectives of transpilers is to make your code compatible for browsers that do not necessarily support ES modules. We should remember though that our libraries are not meant to be served to browsers directly but should instead be consumed by applications. So transpiling our libraries to target specific browsers should be forbidden for two reasons:

When making a library, one does not know which browser should be targeted, only the application knows that.
Transpiling libraries can make them non tree shakeable

If the library needs to be transpiled for some reason, one needs to make sure the transpilation does not remove the ES module syntax or the original module tree for the same reasons explained in the last part of the article.

There are two tools that I know of that transpiled my libraries removing their tree shakeable characteristic.

Babel

Babel uses Babel preset-env to make your code compatible with one's target browsers. By default, this plugin will remove ES modules from the library. To make sure this does not happen, set the modules option to false:

module.exports = {
  env: {
    esm: {
      presets: [
        [
          "@babel/preset-env",
          {
            modules: false,
          },
        ],
      ],
    },
  },
};

Typescript

When compiling your code, typescript will transform your modules depending on the target and module options you set in the tsconfig.json file.

To make sure this does not happen, set the target and module options to at least ES2015 or ES6.

To sum up:

Make sure your transpilers/compilers do not remove the ES module syntax from your library bundle.
To check whether this happens, look at the library's output bundle and check for ESM import syntax.

Use the latest version of a tree shaking capable bundler

Tree shaking in javascript was popularized by Rollup. Webpack supports this since version 2 and bundlers keep getting better and better at optimizing tree shaking.

Remember when we talked about the innerGraph optimization that allows Webpack to link module exports to the module's imports? This optimization was introduced in Webpack 5. We have been using Webpack 5 in this article but it is important to note that this optimization is a game changer as it allows Webpack to recursively look for unused exports!

To show what it actually does, we can consider our index.js file in our user-library:

import { userAccount } from "./userAccount";

const getUserAccount = () => {
  return userAccount;
};

const getUserName = () => "John Doe";

export { getUserName, getUserAccount };

Our user-app only uses getUserName.

import { getUserName } from "user-library";

console.log(getUserName());

We can now compare the outputs with and without the innerGraph optimization. We are still using both the usedExports and sideEffects optimizations:

Without the innerGraph optimization (eg with Webpack 4):

/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/*! exports provided: userAccount, userPhoneNumber, getUserName, getUserAccount */
/*! exports used: getUserName */
/***/ (function(module, __webpack_exports__, __webpack_require__) {

"use strict";
/* harmony export (binding) */ __webpack_require__.d(__webpack_exports__, "a", function() { return getUserName; });
/* unused harmony export getUserAccount */
/* harmony import */ var _userAccount_js__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(/*! ./userAccount.js */ "./node_modules/user-library/dist/userAccount.js");

const getUserAccount = () => {
	return _userAccount_js__WEBPACK_IMPORTED_MODULE_0__[/* userAccount */ "a"]
};

const getUserName = () => 'John Doe';

/***/ }),

/***/ "./node_modules/user-library/dist/userAccount.js":
/*!*******************************************************!*\
  !*** ./node_modules/user-library/dist/userAccount.js ***!
  \*******************************************************/
/*! exports provided: userAccount */
/*! exports used: userAccount */
/***/ (function(module, __webpack_exports__, __webpack_require__) {

"use strict";
/* harmony export (binding) */ __webpack_require__.d(__webpack_exports__, "a", function() { return userAccount; });
const userAccount = {
	name: 'user account'
};

/***/ }),

With the innerGraph optimization (eg with Webpack 5):

/***/ "./node_modules/user-library/dist/index.js":
/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {

/* harmony export */ __webpack_require__.d(__webpack_exports__, {
/* harmony export */   "getUserName": () => (/* binding */ getUserName)
/* harmony export */ });
/* unused harmony export getUserAccount */

const getUserAccount = () => {
	return userAccount
};

const getUserName = () => 'John Doe';

/***/ })

While Webpack 5 is able to completely eliminate the userAccount module, this is not the case for Webpack 4 even though getUserAccount is marked as unused. This is because the innerGraph algorithm allows webpack 5 to link unused elements of our module with its imports. In our case, the userAccount module is used only by the getUserAccount function and can therefore be skipped.

This optimization does not work using Webpack 4. Developers should therefore be careful and limit the number of exports in a single file when using this version of Webpack. I a file contains multiple exports, Webpack will include all the file imports even though they may not be necessary for the desired export.

In general, we should make sure our bundlers are always up to date to benefit from their latest tree shaking optimizations.

Conclusion

Tree shaking a library is not something one just turns on by adding a specific line in a configuration file. Its quality depends on multiple factors and this article presents only a few of them. In the end though, whatever the issue might be, there are two important things we did in this article that can help anyone tree shake their libraries:

In order to see how well our library is tree shaked, we test it in a controlled environment using a bundler we know how to use.
We detect issues with our library setup not by looking at its configuration files but by inspecting its bundled output. This is what we have been doing all along in this article with our user-library and user-app examples.

I really hope this article helps you in your evergoing quest to make the best and most optimized libraries possible!

An introduction to DynamoDB data modeling

Mon, 12 Apr 2021 00:00:00 GMT

I discovered DynamoDB a few months ago, in the context of a very high traffic web application built using the Serverless Framework. DynamoDB - as well as Single-Table design - was fairly new to me, and I quickly noticed that there are a few concepts to grasp before designing the first data models. This article aims to provide a beginner-friendly tutorial for developers who are considering starting a project with DynamoDB, or who are just curious about this technology.

DynamoDB is a fully managed NoSQL database released by AWS in 2012. As the other main NoSQL solutions, such as MongoDB (2009) or Apache Cassandra (2008), it was designed at a time when storage prices were low compared to those of processing power. In the seventies, when relational databases were first introduced, this relationship was inverted. In particular, high storage prices gave incentives to limit storage needs by avoiding data duplication. You would design separate entities and put each one of them in a separate table, which drastically simplifies the data modeling exercise. You could then simply write the entity-relationship diagram (ERD), translate it into tables and retrieve the information of your choice by joining tables together.

With a NoSQL database such as DynamoDB, the join functionality does not exist, you're invited to put all your entities in the same table. This is the concept of Single-Table Design. By removing joins, you write more efficient queries, which reduces the needs of computing power. However, this comes at a cost: the data modeling exercise is slightly more complicated. The entity diagram does not give you the model anymore. The ways your application will interact with data - the access patterns - are now a key building block of your modelisation.

In this tutorial, I will take the example of a sample book review website. On the application, users can write a review for a book of their choice. They can also read the reviews written by others on any given book. Finally, users can read all the reviews written by a given user. As a bonus, we will talk about sorting these reviews in ante-chronological order.

Start from an entity-relationship diagram

The starting point of the exercise is the ERD. We won't need any joins here, which are the reflection of relationships between entities, but having these relationships in mind is necessary to know how you'll make it work without them.

In our case, the diagram is quite simple. There are 3 entities: books, users and reviews. There is a one-to-many relationship between users and reviews (a user can write several reviews), and another one-to-many relationship between reviews and books (a book can be reviewed several times). In this case, reviews implicitly represent the many-to-many relationship between users and books.

Entity-relationship diagram for users, books, and reviews.

As described on the diagram, we use the ISBN (International Standard Book Number) as a unique identifier for books, and the username for users. A review consists of a date, a grade, and a text.

Write down the access patterns

Before any attempt to model the keys in your table, you should write down the access patterns of your application. The modelisation's purpose is to handle all of them. It is important to be thorough in this step, or you might be unable to retrieve some information from your table. For the purpose of this exercise, let's consider three of them.

Access Pattern
Get all the reviews for a given book
Get all the reviews for a given user
Update a review

On the book page, we need the reviews for this particular book. On a user profile, we will query for the reviews written by this user. Finally, we will see how we would update a review.

Model the structure of the primary key

Once you have defined all of your access patterns, you can start the modelisation. The primary key is a core concept in DynamoDB, and almost the single piece of structure imposed to your table. It is thus at the core of the modeling exercise.

DynamoDB allows two kinds of primary keys:

Simple primary keys, made of a single element called partition key (PK).
Composite primary keys, made of partition key (PK) and sort key (SK).

The partition key is particularly important, as it represents an identifier of the server on which the data actually is (also called a storage node). Two items that have distinct partition keys will thus be difficult to query together.

The primary key related rules are simple:

A table must have a primary key
All the items in the table must have a value for that primary key
Each primary key must be unique <> an item is uniquely identified by its primary key
Each request to READ item(s) must specify the partition key *
Each request to WRITE an item must specify the full primary key

* Here I intentionally avoid scans, which are 9 times out of 10 bad practice

With these concepts in mind, you can start modeling. For each entity, we have to think about the best partition key and sort key to fulfill all the access patterns. In other words, the goal of this step is to fill the table below:

Entity	PK	SK
Book	`?`	`?`
Review	`?`	`?`
User	`?`	`?`

A common practice is to start with the top-level entity that has the highest number of relationships. Here, books and users play a somehow symmetric role so let's start with modeling books.

The goal is to be able to query at the same time for a book and its associated reviews. We thus have to make sure that the book reviews have the same partition key as the book itself, in order for them to be stored on the same physical machine. Using this partition key, DynamoDB will identify the associated storage node in a O(1) time. In DynamoDB terminology, we say that we put the book and its reviews in the same item collection.

Because an item in DynamoDB must be uniquely identified by its primary key, the sort key will be the way we differentiate a book from its reviews. Let's have a look at the following primary key structure:

Entity	PK	SK
Book	`BOOK#<ISBN Number>`	`BOOK#<ISBN Number>`
Review	`BOOK#<ISBN Number>`	`USER#<Username>`

Prefixing the key by the entity name is a common pattern. It gives us more clarity and compensates for the fact that we put all entities in the same table. The ISBN of the book is here used as a unique id. It enforces the uniqueness of a book's primary key. Finally, the sort key USER#<Username> allows us to differentiate reviews. It also identifies the author of the review. Implicitly, it says that a given user can not write two reviews about the same book (otherwise it violates the primary key's uniqueness principle).

Visualize your model in NoSQL Workbench

During the modeling exercise, it is good practice to model out some example data. It especially helps to visualize how a query would respond. The NoSQL Workbench is a great tool for that. At this stage of the modelisation, here is how a simple example would look like:

Main index - Books and reviews visualisation in NoSQL Workbench.

A row in this table is an item in DynamoDB. Both items share a common partition key, they are in the same item collection and can thus be queried together. We choose generic names for the key attributes (PK and SK) because they do not store the same type of information depending of the kind of entity they belong to.

To solve the first access pattern, just make a request for the items with a PK of BOOK#<ISBN>, with no condition on the sort key (remember that for a READ operation, specifying the sort key is optional). Both items in the table match this condition, so they will be returned together, sorted alphabetically on the sort key. In JSON format, you get something like

[
  {
    "PK": {
      "S": "BOOK#<ISBN>"
    },
    "SK": {
      "S": "BOOK#<ISBN>"
    },
    "Author": {
      "S": "J.R.R. Tolkien"
    },
    "Title": {
      "S": "The Fellowship of the Ring"
    },
    "PublicationYear": {
      "N": "1954"
    }
  },
  {
    "PK": {
      "S": "BOOK#<ISBN>"
    },
    "SK": {
      "S": "USER#<username>"
    },
    "Grade": {
      "N": "10"
    },
    "ReviewText": {
      "S": "Must read."
    },
    "ReviewDate": {
      "S": "2001-08-05T02:46:00"
    }
  }
]

Define secondary indexes if needed

Let's move on to the second access pattern. We want to be able to query for a user and all their reviews. We therefore need the user to be in the same partition as their reviews. But these already share a common partition key with the book they relate to. We thus somehow need the reviews to be in two distinct item collections. This is the notion of global secondary index (GSI).

When creating a GSI, you specify two attributes in your table that will act as a primary key in the secondary index. In other words, this index will be a copy of the data of the main table, in which the primary key is different. Using this, we can have the book and reviews to share a common PK in the main index, while having the reviews and the associated author in the same partition in a secondary index.

In our case, we can then adopt symmetric approaches in the main and secondary index. In the main index, the primary key structure will be like below:

Entity	PK	SK
Book	`BOOK#<ISBN Number>`	`BOOK#<ISBN Number>`
Review	`BOOK#<ISBN Number>`	`USER#<Username>`
User	`USER#<Username>`	`USER#<Username>`

See that the primary key structure for books and users follow the exact same pattern. In NoSQL Workbench, adding the review author to our example would bring us to:

Main index - Users, books and reviews visualisation.

Notice that both the user and their review have a SK of USER#<Username>. Having the SK of the main index as partition key in the secondary one would therefore solve the second access pattern. More precisely, we can just flip the PK and SK of the main index in the GSI. The secondary index in NoSQL Workbench would then be:

Secondary index - Users, books and reviews visualisation.

To solve the second access pattern, we make a request on the secondary index for the items with a partition key (here the SK attribute) of USER#<Username>, with no condition on the sort key. Both the user and the review match this condition, so they will be returned together in an array, sorted alphabetically on the sort key. To retrieve the user first, we can set the scanIndexForward property to false, so that DynamoDB reads the items in descending order. Note that by default (with scanIndexForward property set to true), DynamoDB will send the items in the order they are sorted in the table, which is the ascending order. If you are unsure, just look at your items in the NoSQL Workbench: top to bottom is the ascending ordering used by DynamoDB.

At this stage of the modelisation, most of the work is done. Inserting a review - the third access pattern - is trivial. We use the UpdateItem action of DynamoDB (you can learn more about writing items here), by specifying BOOK#<ISBN> for the partition key and USER#<Username> for the sort key (remember that for WRITE operations you must provide both the partition key and the sort key), along with the other attributes characterizing the review - grade, review text and date. By doing this, you implicitly specify a value for the PK and SK in the secondary index, and DynamoDB will thus take care of updating that item in the GSI as well.

What about sorting?

Sorting the query results before they are sent back to your application can be crucial in some situations. Going back to the book review example, imagine that you want to paginate the reviews for a particular book. More precisely, what if you want to show the 1 to X most recent reviews on the first page, then the X+1 to 2X most recent ones on the second page etc... Without sorting, you would have to retrieve all reviews on each page, and then get rid of those that are not in the appropriate range for that page. This would be very inefficient, especially if your application has a lot of contributors.

Let's cover how we would sort the reviews in ante-chronological order. As we have seen so far, in DynamoDB, almost everything is achieved using the primary key. It is also true for sorting. If you want a special ordering of the items returned from a query, there is only one way to do so: it has to be done with the sort key. Implicitly, it means that you need to arrange your items so that they are sorted in advance. This again stresses the need for thorough data modeling upfront.

In our case, the sort key thus needs to cary some sort of time indicator. Both unix timestamps (e.g. 744569696) or ISO-8601 date format (e.g. 2021-03-05T18:54:56+00:00) would work. The only requirement is that it must be sortable. Let's go for ISO- 8601 timestamps as they are human-readable. We can then prefix the sort key for reviews like so:

Main index - Timestamped version - Users, books and reviews visualisation.

For the review entity, the partition key and sort key in the secondary index are no longer an inversion of the primary key in the main table. That's why we have to add specific attributes to our table. We name them generically GSIPK and GSISK respectively as they do not store the same information per entity. They will act as partition and sort key in the secondary index.

To get a book and the associated reviews sorted, the query is the same as before. Query all the items with a partition key of BOOK#<Username> with no restrictions on the sort key. With the scanIndexForward property set to false, you would retrieve the book and then the review. As DynamoDB orders the items in ascending orders in your table, a more recent review would come after the first one:

Main index - Timestamped version - Sorted reviews.

With the scanIndexForward property set to false, you retrieve the items from bottom to top. This gives you the book and then the reviews from most recent to oldest. Finally, use the limit option (like in SQL) set to 16 for example and you restrict yourself to the 15 most recent reviews.

In the secondary index, the data is structured symmetrically. In this index, the item attributes GSI1PK and GSISK are the partition and sort key, respectively.

Secondary index - Timestamped version - Sorted reviews.

A query for items with a partition key (here the GSIPK attribute) of USER#<Username> with no restriction on the sort key gives us the user and their reviews. With scanIndexForward property set to false, the user will come first, and then their reviews in ante-chronological order. The response, in JSON format would be:

[
  {
    "PK": {
      "S": "USER#<Username>"
    },
    "SK": {
      "S": "USER#<Username>"
    },
    "MembershipDate": {
      "S": "2000-09-29T04:31:00"
    },
    "GSI1PK": {
      "S": "USER#<Username>"
    },
    "GSI1SK": {
      "S": "USER#<Username>"
    }
  },
  {
    "PK": {
      "S": "BOOK#<ISBN>"
    },
    "SK": {
      "S": "2002-08-05T02:46:00#USER#<Username>"
    },
    "Grade": {
      "N": "5"
    },
    "ReviewText": {
      "S": "Fantastic"
    },
    "ReviewDate": {
      "S": "2002-08-05T02:46:00"
    },
    "GSI1PK": {
      "S": "USER#<Username>"
    },
    "GSI1SK": {
      "S": "2002-08-05T02:46:00#BOOK#<ISBN>"
    }
  },
  {
    "PK": {
      "S": "BOOK#<ISBN>"
    },
    "SK": {
      "S": "2001-08-05T02:46:00#USER#<Username>"
    },
    "Grade": {
      "N": "5"
    },
    "ReviewText": {
      "S": "Must read."
    },
    "ReviewDate": {
      "S": "2001-08-05T02:46:00"
    },
    "GSI1PK": {
      "S": "USER#<Username>"
    },
    "GSI1SK": {
      "S": "2001-08-05T02:46:00#BOOK#<ISBN>"
    }
  },
]

Wrapping it all up

In this simple modeling exercise, we have set out the key steps that should be followed when creating a data model in DynamoDB. Write your entity diagram to identify the relationship between objects in your application. Carefully write down your access patterns and start modeling the structure of primary keys for your entities. Try to do your best with the main index, and then make the use of secondary indexes if needed. During the exercise, take a practical example and visualize it in the NoSQL Workbench.

Once done, celebrate, as scaling is not a problem anymore. In our example, a new book would represent a new partition, and therefore a new storage node: DynamoDB abstracts the complexity of database scaling from you. Because partitions are split up on different physical machines, and because the time to find the partition you query for is constant, your queries efficiency will remain stable as the amount of data increases.

However, this automatic scaling isn't free - the modelisation exercise is slightly more complicated than with relational database systems. You can't just go from your entities diagram to the data model. Access patterns must be thought of in advance, as they should drive the modelisation.

Because of this, there are a few situations in which DynamoDB might not be the optimal choice. If your application is meant to allow for a virtually unlimited number of access patterns - an analytics platform for instance - then DynamoDB will not be a good fit. Or, if you are building a new application from scratch, you may not know how its functionalities will look like in the longer run. It will then be tough to think of all the access patterns in advance. Finally, keep in mind the learning curve. If you plan to build a product as fast you can, and if your team is much more familiar with relational database systems, you might want to stick to it to maximize development speed.

If not in one of these situations, go and try DynamoDB. To speed up learning, I would strongly recommend reading The DynamoDB Book by Alex Debrie. You'll find plenty of well chosen examples as well as the main theoretical concepts behind it. Enjoy!

Icon Library in React: Why Inline SVG Are Better than a Font

Thu, 25 Mar 2021 00:00:00 GMT

How I managed to choose a performant icon system adapted to a Next.js project with styled-components.

Almost every website uses icons in order to quickly guide users or to improve the aesthetics of the page.

Usually, designers can export icons to SVG format. SVG stands for Scalable Vector Graphics which means they can grow indefinitely, as they are defined by a description, not by pixels like in pictures.

In a new Next.js/React project with a hundred icons, I had to choose the best way to integrate icons. In this article, I propose to:

Share a bench of four methods to integrate SVG
Compare the best promising methods: inline SVG and font
Detail the implementation of the retained method

So many ways to integrate SVG icons!

Once you have an SVG (let's say a wonderful arrow icon), there are at least four ways to import them... not all of these methods are worth it!

SVG as an image

An SVG can be used as the source of an img HTML tag:

const Icon = ({ src, ...imgProps }) => <img src={src} {...imgProps} />;

const ArrowIcon = <Icon src="icons/arrow.svg" alt="Big arrow" />;

Even if this icon can be manipulated exactly as an image, advantages of SVG, except file size reduction, cannot be used. In particular, customization is not possible. Moreover, if the file is moved or deleted, developers may not be aware of it. Let's see what's next!

SVG as a background image

A really simple way to create an icon is to set it as the background image of a div element.

import styled from "styled-components";

const Icon = styled.div`
  ${({ url }) => `background-image: url(${url});`}
  display: flex;
  align-items: center;
  justify-content: center;
  overflow: hidden;
  background-size: cover;
  background-position: center center;

  ${({ size }) => `
    width: ${size}px;
    height: ${size}px;
  `};
`;

const ArrowIcon = <Icon url="icons/arrow.svg" size={20} />;

The icon can be manipulated as a div element, so the size is customizable, but that's about it. Thus, let's go through another way.

SVG as font

One popular method to manage a big set of icons is to use a font. It is quite easy to generate a font with online tools such as IcoMooon or Fontello.

A generic icon component should look like:

import "./style.css";

const Icon = (props) => (
  <span>
    <i className={`icon-${props.slug}`} />
  </span>
);

Where the file style.css, normally created by a font generator, should be something like:

@font-face {
  font-family: "icons";
  src: url("fonts/icons.woff2") format("woff2"), url("fonts/icons.woff") format("woff");
  font-weight: normal;
  font-style: normal;
  font-display: block;
}

.icon-arrow:before {
  content: "\\e954"; /* `\\e954` is a character defined in the font */
}

For additional props, you can wrap the icon, replacing span by a styled component with props. For instance:

const WrappedIcon = styled.span`
  ${(props) => `
    i {
      ${props.color ? `color: ${props.color};` : ""}
      ${props.size ? `font-size: ${props.size}px;` : ""}
    }
  `};
`;

And finally get a simple icon component:

const ArrowIcon = <Icon slug="arrow" size={20} color="blue" />;

This method seems to be used in a lot of projects. Before challenging it, let's see the last one.

Inline SVG

Importing icons in React (granted it is properly configured, we'll see that later) is as simple as that:

import Arrow from "icons/arrow.svg";

const MyComponent = () => <Arrow />;

This is the method I chose in my case, let's understand why.

Comparison

Criteria ✅: OK ⚠️: Points of attention ⛔: KO if it is a regarded criterion	Inline SVG	Font	SVG as image	SVG as background image
Customization	✅ Highly customizable: `size`, `color`, etc. and specific properties `fill` and `stroke`	⚠️ Customizable like text only: `size`, `color`, etc.	⛔ Limited customization, same as image: `size`, etc.	⛔ Limited customization, via its div container: `size`, etc.
Customization	✅ All parts of the SVG can be customized finely	⚠️ Painful positioning as it uses text positioning CSS	⛔ Limited customization, same as image: `size`, etc.
Performance	✅ Smaller files compared to images	✅ Smaller files compared to images	✅ Smaller files compared to images	✅ Smaller files compared to images
	⚠️ Icons are loaded with the DOM: it may slow the page loading, but ensures all icons are loaded at the same time as the page	✅ Can be loaded thanks to a CDN	⚠️ Icons are loaded with the DOM: it may slow the page loading, but ensures all icons are loaded at the same time as the page	⚠️ Icons are loaded with the DOM: it may slow the page loading, but ensures all icons are loaded at the same time as the page
		⚠️ Font is loaded in parallel: blank characters can appear. All icons are loaded even if few are used
Maintainability	✅ Easy to add icons progressively	⚠️ Painful to regenerate a font, even if some external tools like Icomoon or Fontello can help	⛔ Difficult to assert images path are valid	⛔ Difficult to assert images path are valid
Developer Experience	✅ Good experience if SVG files are easy to customize	✅ Good experience once the font is configured	✅ Good experience	✅ Good experience
Accessibility	✅ Possibility to define `title` and `desc` tags	⛔ Not accessible without large efforts	✅ Accessible like images	⛔ Not accessible

Let's dive further into the comparison of the two concurrent methods: inline SVG vs. font...

Customization

Fonts are managed like text and it is therefore possible to customize an icon's size or color. Precise positioning is possible with the same properties as text (font-size, line-height, text-align, etc.), so it is quite painful when icons are used as buttons or images. For emojis used in a text, it could be more relevant, however.

SVG are customizable as images and properties fill and stroke allow to redefine colors respectively of the content or the border. All parts of the SVG can be separately customized, which is impossible to do with fonts.

➡️ Using inline SVG is the best choice if you need color variants of icons, or to customize precise parts of them.

➡️ Icons used as characters should be part of a font.

Performance

Fonts are loaded asynchronously, which means blank characters can appear while the font is being loaded.

All icons are loaded in the font, even if only a few of them are used on the page. However, it can be served by a CDN, a fast method if CORS is correctly configured.

On the contrary, SVG are loaded with the DOM, which avoids blank spaces, but relatively increases loading duration. An icon usually weighs from 500B to 5kB (less with SVG Optimization) so it can be non-negligible if hundreds of icons are used at a time - quite rare, however!

Fonts are also subject to anti-aliasing methods applied by browsers to font characters when scaling. SVG do not suffer from this issue and are fully scalable.

➡️ In pages with hundreds of icons, using a font reduces the page size. Serving it via a CDN is a performant option.

➡️ For pages with moderate use of icons, SVG is a better option to avoid anti-aliasing and blank characters at loading.

Maintainability

What if icons evolve along the development process?

With a font, you need to regenerate the font correctly, with the correct source files - otherwise, some character codes may change!

With SVG, you just need to add, replace or delete a file and update file imports in the project. In case of replacement, check the SVG structure is still the same, as fine customization of SVG parts may be affected.

➡️ If icons are likely to evolve, SVG files are a more consistent and independent format to handle icon addition, edition, or deletion.

Developer experience

Once icon systems are set, both methods are easy to use as developers, as seen in the first part.

The only point of attention can be when it is necessary to customize SVG icons that are already styled. For instance, with a fill or a width property already set:

<svg width="15px">
  <path fill="none" d="..." />
</svg>

In this case, applying custom fill or width won't be applied. Cleaning the SVG by removing fill="none" and width="15px" should fix the issue.

➡️ Once an icon system is set, the developer experience should be satisfying in both cases.

Browser compatibility

Fonts are supported by almost all browsers, using the @font-face rule. All modern web browsers support WOFF2 and WOFF which are optimized formats for the web, but other formats can be used for broader browser support. However, Opera Mini does not support @font-face at all.

All modern browsers now support inline SVG method, even Internet Explorer which only does not support CSS transforms (more details on caniuse website).

➡️ Both methods are supported by more than 98% of web browsers. Inline SVG is slightly better supported and has better fallback options.

Accessibility

Serving accessible fonts is hard. If you are adventurous, this article details the tricks to guarantee accessible fonts, so I won't expand on it here.

If accessibility is important for your project, then SVG are made for this. An SVG file may have a title or even a desc (for description) tags. For better browser support, it is possible to add an id to these tags and refer to them the attributesaria-labelledby as the following minimal example shows.

<svg aria-labelledby="circleTitle circleDescription" role="img">
  <title id="circleTitle">A circle</title>
  <desc id="circleDescription">A red circle with a blue border</desc>
  <circle cx="50" cy="50" r="40" stroke="blue" stroke-width="2" fill="red" />
</svg>

For more details, I advise you to look at this CSS-tricks page.

➡️ For accessible projects, SVG is the best option with no hesitation.

A winner?

A font system is an option to consider if your project uses a huge amount of icons per page, is focused on performance, does not need to be accessible, and has a graphic chart that won't evolve.

In other cases, for better flexibility and user experience, I would recommend an inline SVG system. That's what I chose in the case of a new B2C project with common use of icons (<30 per page) and a library with possible evolutions.

In the following part, I will explain how to configure such an icon system.

Configuring an SVG icon system with Next.js, React and styled-components

Here are the four simple steps I followed to integrate icons in a Next.js/React project using styled-components as styling library.

Install babel-plugin-inline-react-svg:

# With npm
npm install --save-dev babel-plugin-inline-react-svg
# or, with yarn
yarn add --dev babel-plugin-inline-react-svg

This package will allow the import of SVG files as React components. It also uses SVGO to optimize SVGs.

Note: SVGR also offers good alternatives to babel-plugin-inline-react-svg depending on your bundler, such as @svgr/webpack.

Add inline-react-svg plugin to babel configuration (babel.config.js or .babelrc):

plugins: ["inline-react-svg"];

For further configuration, refer to the npm page.

In a TypeScript project, you should also add a module declaration in a type file:

// fileTypes.d.ts

declare module "*.svg" {
  import React = require("react");
  const src: (props: React.SVGProps<SVGSVGElement>) => JSX.Element;
  export default src;
}

Now, it is possible to import directly SVG as React components. For a resilient icon system, it is possible to create a JSX/TSX file per icon:

// icons/Arrow.jsx
import Arrow from "./arrow.svg";

// ... (possible customization of the component)

export default Arrow;

With this method, refactoring icons is easier if the import method changes.

Warning: using a unique index file with all SVG icons gathered in it is not recommended, because it prevents from tree-shaking the library.

Import SVG files as React components:

import Arrow from "icons/Arrow";

const MyComponent = () => <Arrow />;

It is also possible to customize icons with styled-components, as shown in this example:

// MyComponent.tsx
import styled from "styled-components";
import Arrow from "icons/Arrow";

const CustomSpan = styled.span`
  color: red;
  font-size: 40px;
`;

const CustomArrow = styled(Arrow)`
  width: auto;
  height: 30px;
  stroke: blue;
  stroke-width: 50px;
  fill: currentColor; /* fill with the current color, here red */
`;

const MyComponent = () => (
  <button>
    <CustomSpan>
      Go <CustomArrow />
    </CustomSpan>
  </button>
);

...which renders as a wonderful button:

That's all you need to manage icons in your project!

File Download with Token Authentication and Javascript

Wed, 24 Mar 2021 00:00:00 GMT

Recently on a project of mine, I was presented with a problem: I needed to allow our users to download a ZIP archive of files from our back-end, based on complex authorisation rules, following a four-step process:

The front-end would call the back-end with a list of file names to download.
The back-end would authenticate the user with their JWT token.
It would then check if the user was allowed to retrieve those files.
Finally, it would send them the archive back.

Up until now, we were relying on ‘security through obscurity’, meaning we were trusting the uniquely generated file names to prevent users from snooping around and downloading files they were not supposed to retrieve.

But this was not enough: how could we revoke the rights of the user on a file? How could we be certain that a user would not be accessing a file they were not supposed to access? We needed to add authentication to the route.

The Unauthenticated Case

First, let me show you the code we were using until now. In this case, the back-end is a Spring Boot microservice, and the front-end is an Angular SPA.

Below is a simplified version of the unauthenticated case. What this does is define a GET route /files/download taking a set of file IDs as parameters, and retrieving, zipping and sending the files back to the client.

@GetMapping("/files/download")
public ResponseEntity<ByteArrayResource> download(
    @RequestParam Set<UUID> fileIds
) {
    ByteArrayResource zippedFiles = new ByteArrayResource(
        fileService.getZippedFiles(fileIds)
    );

    return ResponseEntity
        .ok()
        .contentLength(zippedFiles.contentLength())
        .header(HttpHeaders.CONTENT_TYPE, "application/zip")
        .header(
            HttpHeaders.CONTENT_DISPOSITION,
            ContentDisposition
                .builder("attachment")
                .filename("archive.zip")
                .build()
                .toString()
        )
        .body(zippedFiles)
    ;
}

So now, if you make a GET request to /files/download?fileIds=UUID, you’ll receive the file with name UUID zipped. This allows you to keep your front-end code extremely simple because a simple anchor element pointing to the URL to the back-end route will be enough to trigger a download pop-up in the client’s browser, thanks to the Content-Disposition header.

So your front-end component could retrieve the required file names, concatenate them somehow, and then simply display a link to the back-end URL, like so:

<a
  href="{{ backendUrl }}/files/download?fileIds={{ fileNames$ | async }}"
>Download your archive</a>

This is all good and well, but when you do this, you cannot forward JWT tokens, so you cannot authenticate your user: you need to download the files some other way.

Adding Authentication to the Route

So first, you need to add some kind of authentication and authorization. Though my project required complex authorization rules, let us say you only want to be sure the user is logged in before sending them the files. This is only a matter of adding a decorator to your back-end route:

@GetMapping("/files/download")
@PreAuthorize("hasRole(T(fr.theodo.security.Roles).USER)")
public ResponseEntity<ByteArrayResource> download(…) {…}

This is the easy part. Now, if you were to attempt using the aforementioned link, you would get an error: as no authentication token is provided, the back-end responds with a 401 Unauthorized response. How do we fix that?

Retrieving the File with JavaScript

Downloading the file will be done in two steps: first, you will download the file using JavaScript, allowing you to set the authentication token, then, you will ‘forward’ the file to your user.

Alternatively, note that you could set up a different authentication method specifically for this route, such as a cookie-based one, though this would mean having two different authentication methods on your project, with one being used by a single route… so I would advise against it.

Downloading the File

Assuming you already perform authenticated calls to your back-end using some kind of API client, downloading the file will be straightforward: you will instantiate your client in your component, and call your back-end (here, it is done with this.apiClient.downloadZipFile).

The switchMap RxJS operator allows you to discard any previous calls if this.project$ changes; you can read more about it on Learn RxJS.

export class MainPanelComponent {
  @Input() project$!: Observable<Project>;

  constructor(private apiClient: ApiClientService) {}

  private getZip(): Observable<string> {
    return this.project$.pipe(
      switchMap(project => this.apiClient.downloadZipFile(project.files)),
    );
  }
}

‘Forwarding’ the File to Your User

Now that you have retrieved your file, returned as an Observable by switchMap, you need to trigger the save on your user’s computer. This is only a matter of chaining two new operators in the RxJS pipe:

…
return this.project$.pipe(
  switchMap(project => this.apiClient.downloadZipFile(project.files)),
  map(data => window.URL.createObjectURL(data)),
  tap(url => {
    window.open(url, '_blank');
    window.URL.revokeObjectURL(url);
  })
);
…

First, you map the retrieved data to an object URL (a simple string). MDN will explain them in greater details, but the main things you need to know are that they are a way to create local URLs pointing to the browser memory… and that they are not supported by Internet Explorer, so you may want to keep that in mind depending on your target audience.

Finally, you tap into this new Observable to expose the file to your user (simply opening a new window with this local URL will work) and clean up behind yourself by removing the object URL (this allows the browser to free up some memory once the file is downloaded).

Triggering the Download

You may have noticed that for now, you have a getZip method, but you do not call it. To remedy this, you only need two new lines of code in your component and one in its template:

export class MainPanelComponent {
  @Input() project$!: Observable<Project>;

  zipDownloadTrigger = new EventEmitter<void>();

  constructor(private apiClient: ApiClientService) {
    this.zipDownloadTrigger.pipe(exhaustMap(this.getZip)).subscribe();
  }

  …
}

<button (click)="zipDownloadEmitter.emit()">Download your archive</button>

This defines an event emitter, which emits after each click on a button. You chain this emitter with an exhaustMap RxJS operator, which will call the getZip method, and wait for it to complete before resuming listening to the event emitter (hence the ‘exhaust’; again, refer to Learn RxJS for more details).

This means that all clicks on the button are ignored while the component is downloading the file. Note that you could do without this, in which case you could simply pass the getZip method as a click callback on the button.

Note that for the sake of simplicity, I use a button here, but this is a bad practice in terms of accessibility: you should always use a download link to download a file. Here you could have a button to trigger the download, then display the link with the local URL as its href attribute.

UX Niceties and the Final Code

If you put all this together and add a simple loading toggle, you get the following code:

export class MainPanelComponent {
  @Input() project$!: Observable<Project>;

  isTheArchiveLoading: boolean = false;
  zipDownloadTrigger = new EventEmitter<void>();

  constructor(private apiClient: ApiClientService) {
    this.zipDownloadTrigger.pipe(exhaustMap(this.getZip)).subscribe();
  }

  private getZip(): Observable<string> {
    this.isTheArchiveLoading = true;

    return this.project$.pipe(
      switchMap(project =>
        this.apiClient
          .downloadZipFile(project.files)
          .pipe(finalize(() => { this.isTheArchiveLoading = false; }))
      ),
      map(data => window.URL.createObjectURL(data)),
      tap(url => {
        window.open(url, '_blank');
        window.URL.revokeObjectURL(url);
      })
    );
  }
}

This ensures that you can show a loader to your user if isTheArchiveLoading is true. The finalize operator in the pipe on downloadZipFile is called when downloadZipFile resolves, much like the Promise.finally() method.

And here you go, you have a fully functional downloading method, with an authenticated route, called from your front-end!

Improve accessibility on your Angular Apps with Jasmine and Axe

Mon, 22 Mar 2021 00:00:00 GMT

At Theodo, we have multiple projects using Angular and we are currently focusing on building more accessible websites. (learn more about a11y)

We had an experience with Axe and jest-axe on other React projects, and we wanted something similar for Jasmine, the test framework on the Angular project.

What is Axe

Axe is an open source accessibility testing engine by Deque. It can automatically find about 30% of accessibility defects, such as missing labels on inputs, missing alt on images, ...

We found some tests examples with Jasmine in the Axe documentation but it did not seem as straightforward to use as jest-axe. So we decided to solve this problem and make an open source matcher for Jasmine

How to use jasmine-axe

Here is an example on how to use jasmine-axe:

import { TestBed } from '@angular/core/testing';
import { FormsModule } from '@angular/forms';
import { axe, toHaveNoViolations } from 'jasmine-axe';
import { FormComponent } from './form.component';

describe('FormComponent', () => {
  beforeEach(() => {
    TestBed.configureTestingModule({
      imports: [FormsModule],
      declarations: [FormComponent],
    });
    TestBed.compileComponents();
    jasmine.addMatchers(toHaveNoViolations);
  });

  it('should pass accessibility test', async () => {
    const fixture = TestBed.createComponent(FormComponent);

    expect(await axe(fixture.nativeElement)).toHaveNoViolations();
  });
});

Axe will find the violations and suggest solutions to fix them.

A violation was found in a component: Axe suggests some solutions to fix it.

We cannot rely only on those tests for ensuring full accessibility of a website, but it allows us to automate some checks and catch common errors.

We are eager for feedback so try it on your Angular project and tell us if it helped your team improve accessibility!

The React Native Guide I Wish I Had! - Part 1: Environment and Stateless Components

Mon, 22 Mar 2021 00:00:00 GMT

Quick Links

→ Part 1 - focuses on setting up your environment for react native, typescript and redux, as well as creating stateless components for a Todos app. (you are here!)

→ Part 2 - focuses on integrating redux and adding stateful components to your Todos app.

Table of Contents

Project Source
Introduction
Environment Setup
   Technologies
       Homebrew
       Node Version Manager
       Node & NPM
       React Native CLI
   Text Editor
The Todos App
   Creating The App
       Installing Icons
   Folder Structure
   Designing Components
       Styling
       Stateless Components
Resources

Project Source

For reference, the source code for this project can be viewed and downloaded here.

Introduction

During my first week at Theodo, I had a lot to learn about the main technologies used within the company. I focused my attention on learning to build a simple React Native application using Typescript and Redux. This post is aimed at those with a little knowledge of these technologies, guiding you from environment setup, to having a simple working Todos app! I should reiterate that some React, React Native, Typescript and Redux knowledge is assumed. Check out the Resources section for some useful beginner guides!

Environment Setup

The bane of a developer’s life is often related to setting up a working environment for your development. I’m going to outline the steps that I took to fully set up my development environment for this little project. I’ll also be writing this from a Mac OS perspective, but with the resources provided, I’m sure Unix/Windows translation won’t be too difficult!

Technologies

Homebrew

I try to install all the technologies that I need into a central location. This makes using homebrew - a very well maintained package manager for Mac OS - a natural choice.

To install, open a terminal and type:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Node Version Manager

Though this isn’t completely necessary, I like to install Node version manager so that I can have multiple easily manageable versions of Node installed on my Mac at once.

To install:

Open a terminal and type brew install nvm
Create NVMs working directory with mkdir ~/.nvm

Open (or create) your ~/.profile (or ~/.zprofile if using zsh instead of bash) and add:

export NVM_DIR="$HOME/.nvm"
[ -s "$(brew --prefix)/opt/nvm/nvm.sh" ] && . "$(brew --prefix)/opt/nvm/nvm.sh" # This loads nvm
[ -s "$(brew --prefix)/opt/nvm/etc/bash_completion.d/nvm" ] && . "$(brew --prefix)/opt/nvm/etc/bash_completion.d/nvm" # This loads nvm bash_completion

Now every time you log in to your mac, your .profile (or .zprofile) will load NVM and its bash (or zsh) completion. Instead of logging out and logging in, we can invoke this file by running (replace .profile with .zprofile if using zsh) . ~/.profile in your terminal.

Make sure that NVM has been installed properly by typing nvm -v in your terminal.

Note: You can also place this last section in .bashrc (or .zshrc), but .bashrc (or .zshrc) is invoked every time a terminal window is opened. We only need to load NVM once, so we may as well just do it when we login by placing the code in .profile (or .zprofile)!

Node & NPM

We can install Node and NPM using NVM from the last step. Simply run these two commands in a terminal to install and use the latest version of Node (comes with NPM):

nvm install node
nvm use node
Check the installation worked using node -v in a terminal window.

React Native CLI

React Native command-line interface makes it easy to create a new React Native application.

Head over to the React Native setup guide.
Select the ‘React Native CLI Quickstart’ tab.
Do not install Node using homebrew (you should already have it through NVM)!
Install watchman using homebrew.
Follow the ‘Installing dependencies’ steps for both Target OS’s.

Text Editor

You can use whichever text editor you like, but I am using Visual Studio Code, which I highly recommend as it's very customisable!

Here is a list of the extensions that I use:

Many of these are configurable and there are lots of resources online for customisation!

The Todos App

Now that we have all of the necessary technologies installed, let’s make a React Native todos app!

Creating The App

Using the React Native CLI, we can easily create a React Native app from our terminal. Navigate to a directory where you’d like to store the app, then run:

npx react-native init *AwesomeTSProject* --template react-native-template-typescript

You can replace AwesomeTSProject with whatever project name you like. We are also using the Typescript template for our React Native app.

Note: npx will access the current version of React Native CLI at runtime so that you don’t have to install and manage a global version!

Next, navigate into your project directory and run Metro, a javascript bundler that ships with react native:

cd AwesomeTSProject
npx react-native start

Then run your application on iOS (or for android, replace run-ios with run-android):

npx react-native run-ios

Note: React Native version 0.63.4 is unable to install the most recent version of a debugging tool called Flipper on iOS. If you are running into errors involving this module, navigate to /ios/Podfile and replace use_flipper! with use_flipper!({ 'Flipper-Folly' => '2.3.0' }). Then delete the Podfile.lock and, in a terminal, navigate to the /ios directory and run pod install. Now it should be fixed!

Installing Icons

In order to use icon fonts in a React Native app, they must be installed. I use react-native-vector-icons from NPM.

Android Installation

Go here and follow the recommended installation instructions.

iOS Installation

Open Xcode and open the ProjectName/ios/ProjectName.xcodeproj file.
Follow these (manual) instructions
You may need to run pod install in the ios directory before restarting your application.

Folder Structure

I have created a src directory containing all of the source code for my application. The first step is simply to move App.tsx inside this directory and update the import in index.js.

Each of the remaining folders inside src are as follows:

components - all the React components that build up a page live inside this folder (e.g. custom buttons and inputs.
config - all the files for configuration live here (e.g. default styles, colours)
screens - all the screens for the application live here. They are React components that compose many components from the components directory. (Note: there’s only one screen in this application)
store - all things Redux are stored here. Each folder within store denotes a feature. Each feature folder can contain types, actions, selectors and reducers.

I urge you to create these folders now! Don’t worry about the files, we’ll get to those later.

Designing Components

First, I like to think about the smallest components that I’ll need in my application and then build up. We’ll start with a text component.

Although React Native already supplies a Text component, I like to create an AppText component that encapsulates the default styles for my text. This way, we don’t have to repeat any text styling. Simply create a new AppText.tsx file in the components folder and look at the following code.

import React, { FunctionComponent } from "react";
import { Text, TextStyle } from "react-native";

import defaultStyles from "../config/styles";

interface Props {
  style?: TextStyle;
}

const AppText: FunctionComponent<Props> = ({ children, style }) => {
  return <Text style={[defaultStyles.text, style]}>{children}</Text>;
};

export default AppText;

/src/components/AppText.tsx

Just like in React, we create a functional AppText component, passing in its destructured props. Notice that props are typed using an interface (or type) containing all of the passed props. This interface (or type) is then passed to the generic type of the function component. The children prop comes from the FunctionComponent itself. We can also pass optional styles as a prop when using the AppText component, which has the same type as styles that you'd apply on a React Native Text component.

Styling

Styling a component works by setting its style prop to a style object. This can be inline, but it is good practice to create a stylesheet where each key is a separate style object. The style prop can also be given an array of style objects, where the last style object will take precedence if there are any conflicting rules (much like CSS!).

Any component that can be styled, follows the Flexbox styling routine.

Notice that we are using are defaultStyles.text as the first style for the text component. This is imported from ../config/styles. Let’s go and create that file now.

import { Platform, StyleSheet } from "react-native";

import colors from "./colors";

export default StyleSheet.create({
  text: {
    color: colors.dark,
    fontSize: 18,
    fontFamily: Platform.OS === "android" ? "Roboto" : "Avenir",
    fontWeight: "700",
  },
});

/src/config/styles.ts

This file simply exports a StyleSheet object, which can be imported anywhere in the app! Notice how it uses the colors module. This is another configuration file which is just an object storing your colour names.

const colors = {
  black: "#000",
  white: "#fff",
  red: "#de6e66",
  green: "#8bc989",
  light: "#f0f0f0",
  medium: "#6b6b6b",
  dark: "#0c0c0c",
};

export default colors;

/src/config/colors.ts

Now we can test our component works by navigating to src/App.tsx and replacing the code with the following.

import React, { FC } from "react";
import { SafeAreaView, StyleSheet } from "react-native";

import AppText from "./components/AppText";
import colors from "./config/colors";

const App: FC = () => {
  return (
    <SafeAreaView style={styles.container}>
      <AppText style={styles.text} />
    </SafeAreaView>
  );
};

const styles = StyleSheet.create({
  container: {
    alignItems: "center",
    backgroundColor: colors.light,
    flex: 1,
    padding: 20,
  },
  text: {
    color: colors.red,
  },
});

/src/App.tsx

result

Stateless Components

Now that you know the basics of building stateless components, try building these components yourself. Take a look at my source code if you get stuck!

Button.tsx

AppTextInput.tsx

Card.tsx

Before we move on to Redux, it’s worth taking a look at the /src/components/Screen.tsx component. This component encapsulates some logic to ensure that all content in the application is within the safe area of the screen (not underneath the notch). It also adds padding equal to the height of the status bar if we are using Android. We will use this component in the top level of our todos screen in the screens directory.

Part 2: Redux and Stateful Components

Now you're ready to move on to looking at integrating Redux into your Todos app, as well as adding some stateful components!

Click here for Part 2!

Resources

React Tutorial - useful for learning React!
React Hooks - if you don’t know what react hooks are or how to use them, the official docs are great!
React Native Basics - useful for learning React Native!
Redux Essentials and Redux with React - both on the redux official docs, great source of information for redux newbies!
Typescript in 5 mins! - learn the basics of Typescript!

The React Native Guide I Wish I Had! - Part 2: Redux and Stateful Components

Mon, 22 Mar 2021 00:00:00 GMT

This article takes you through a crash course of redux implementation, as well as adding it to your Todos application.

Quick Links

→ Part 1 - focuses on setting up your environment for react native, typescript and redux, as well as creating stateless components for a Todos app.

→ Part 2 - focuses on integrating redux and adding stateful components to your Todos app. (you are here!)

Table of Contents

Redux
   Actions
   Reducers
   Selectors
   Configure and Connect The Store
Bringing It All Together
   Stateful Components
       AddTodo
       TodosList
Wrapping Up
Resources

Redux

Note: Redux Toolkit is now the recommended way of implementing Redux logic. It wraps the core Redux logic, reducing boilerplate code and potential mistakes. This guide doesn't use Redux Toolkit, but gives a thorough introduction to Redux concepts.

I’ll explain redux in the context of how this application has been built. If you have not followed through Part 1 of this guide, it might be easier to have a look there first!

To give an overview - redux maintains a single state that can be updated using event-like actions and reducers, and accessed using selectors.

Actions

We’ll start with the actions that can be dispatched throughout the application (disregarding the all, complete and incomplete filters for now). An action is an object that has a type member and a payload member. The type is what we want to do (e.g. ADD_TODO or DELETE_TODO). The payload can be anything we want (e.g. the todo text, todo id).

With this knowledge, lets create a file in /src/store/todos called types.ts. Here we will define all the possible action types for a todo along with the todo type itself.

export const ADD_TODO = 'ADD_TODO';
export const DELETE_TODO = 'DELETE_TODO';
export const TOGGLE_TODO = 'TOGGLE_TODO';

export interface Todo {
  id: number;
  text: string;
  completed: boolean;
}

interface AddTodoAction {
  type: typeof ADD_TODO
  payload: string
}

interface DeleteTodoAction {
  type: typeof DELETE_TODO
  payload: number
}

interface ToggleTodoAction {
  type: typeof TOGGLE_TODO
  payload: number
}

export type TodoActionTypes = AddTodoAction | DeleteTodoAction | ToggleTodoAction;

/src/store/todos/types.ts

A todo is defined as an object with an id, some text and a completed status.

Each todo action is defined with its respective type and payload (add todo’s payload is the todo text, delete and toggle todo’s payload is the todo id).

Finally we export a new type which is a composition of all the toto action interfaces.

Now we can move on to creating the todo actions.

import { ADD_TODO, DELETE_TODO, TOGGLE_TODO, TodoActionTypes } from './types';

export const addTodo = (todo: string): TodoActionTypes => (
  { type: ADD_TODO, payload: todo }
);

export const deleteTodo = (id: number): TodoActionTypes => (
  { type: DELETE_TODO, payload: id }
);

export const toggleTodo = (id: number): TodoActionTypes =>  (
  { type: TOGGLE_TODO, payload: id }
);

/src/store/todos/actions.ts

Later on, we will dispatch these todo action creators in the necessary components, much like a Javascript event. Each one is simply a function that takes in the necessary information (todo text, todo id) and returns an action object (type and payload). These actions are consumed by a reducer.

Reducers

Reducers are functions that take a state and an action, perform updates on the state and return a new state. These are much like Javascript event listeners (see the connection between actions - dispatchable events - and reducers - event listeners?).

import { ADD_TODO, DELETE_TODO, TOGGLE_TODO, Todo, TodoActionTypes } from './types';

const initialState: Todo[] = [];

let id = 0;

const todosReducer = (state = initialState, action: TodoActionTypes): Todo[] => {
  switch (action.type) {
  case ADD_TODO:
    return [
      {
        id: ++id,
        text: action.payload,
        completed: false,
      },
      ...state,
    ];
  case DELETE_TODO:
    return state.filter(todo => todo.id !== action.payload);
  case TOGGLE_TODO:
    return state.map((todo) => {
      if (todo.id !== action.payload) {
        return todo;
      }
      return {
        ...todo,
        completed: !todo.completed,
      };
    });
  default:
    return state;
  }
};

export default todosReducer;

/src/store/todos/reducer.ts

A reducer can seem very daunting at first, but it’s not so difficult! Let’s work through it.

It’s good practice to always create an initial state. For our todos reducer, the initial state is an empty list of todos.
I also create an id outside of the reducer. This is to be used inside the reducer to auto increment the id of a newly created todo.
The reducer takes two arguments - the current state and the action.

Inside the body of the function, we look at the action type and perform the necessary updates. In the case of ADD_TODO, we return the original state (list of todos) with the new todo prepended.

Selectors

Selectors, as they sound, are functions that extract refined pieces of information from the global state. They are useful to avoid writing duplicate code and integrate very nicely with react hooks!

import { RootState } from '../types';
import { Todo } from './types';

export const selectTodos = (state: RootState): Todo[] => state.todos;

export const selectTodoIds = (state: RootState): number[] => (
  state.todos.map(todo => todo.id)
);

export const selectTodoById = (state: RootState, id: number): Todo | undefined => (
  state.todos.find(todo => todo.id === id)
);

/src/store/todos/selectors.ts

Each selector derives information from the root state. We will define this type shortly, but as you can see, selectTodos function simply takes the todos list from the root state. selectTodoIds maps each todo to its id and returns an array of ids. selectTodoById searches the todos list and returns a todo with a specified id.

Configure and Connect the Store

Now that we have all the building blocks for maintaining todo state, we should create and connect our store to our application. First, let’s define a type for the root state. This type is the definition for our entire state. So if we added another feature, user authentication for example, we would add this as another key in our RootState type.

import { Todo } from './todos/types';

export interface RootState {
  todos: Todo[];
}

/src/store/types.ts

Then we create our store configuration file.

import { Store, combineReducers, createStore } from 'redux';

import { RootState } from './types';
import todosReducer from './todos/reducer';

const rootReducer = combineReducers<RootState>({
  todos: todosReducer,
});

const configureStore = (): Store => {
  return createStore(rootReducer);
};

export default configureStore;

/src/store/configureStore.ts

First we create a root reducer by combining all other reducers using redux’s combineReducers function. For now, this is only the todos reducer. We’ll give our todos reducer a key of todos (as our todos reducer state is a list of todos). Note that we specify the type of our root reducer by passing it as a generic type to combineReducers.

Finally we export a function that returns a redux store. This function makes use of redux’s createStore function, called with our root reducer.

You might ask why we don’t export the invoked createStore itself? This is because we only want to create the store when the application starts, so we export a function to be invoked when the app's root view is rendered.

Bringing It All Together

Navigate back to the /src/App.tsx file. Now we can import our configureStore function and invoke it before we render our application.

import React, {FC} from 'react';
import {StyleSheet, View} from 'react-native';

import {Provider} from 'react-redux';
import TodosScreen from './screens/TodosScreen';
import colors from './config/colors';
import configureStore from './store/configureStore';

const store = configureStore();

const App: FC = () => {
  return (
    <Provider store={store}>
      <View style={styles.container}>
        <TodosScreen />
      </View>
    </Provider>
  );
};

const styles = StyleSheet.create({
  container: {
    backgroundColor: colors.light,
    flex: 1,
  },
});

export default App;

/src/App.ts

I appreciate that we don’t have the TodosScreen component just yet, but that can be ignored for now. Our main focus is the new Provider component. This is a component provided by redux that allows us to pass down the store behind the scenes, making it accessible by any of its children. We pass the store into the Provider through its props.

Stateful Components

There are 4 components left to make.

AddTodo.tsx

TodoItem.tsx

TodosList.tsx

TodosScreen.tsx

We’ll work through AddTodo together, then I’ll let you do the rest! You should create them in this order:

AddTodo
TodoItem
TodosList
TodosScreen (make this component in the /src/screens directory)

AddTodo

AddTodo is a standard react function component.

import React, {useState} from 'react';

import AppTextInput from './AppTextInput';
import Button from './Button';
import Card from './Card';
import {addTodo} from '../store/todos/actions';
import {useDispatch} from 'react-redux';

const AddTodo: React.FunctionComponent = () => {
  const [todoText, setTodoText] = useState('');
  const dispatch = useDispatch();

  const handleSubmit = () => {
    if (!todoText) return;

    dispatch(addTodo(todoText));
    setTodoText('');
  };

  return (
    <Card title="Add New Todo">
      <AppTextInput
        onSubmitEditing={handleSubmit}
        icon="format-list-bulleted-type"
        placeholder="New Todo"
        onChangeText={(text) => setTodoText(text)}
        value={todoText}
      />
      <Button title="Add Todo" onPress={handleSubmit} />
    </Card>
  );
};

export default AddTodo;

/src/components/AddTodo.tsx

We use the react useState hook to maintain a local state of the todo text entered in the input field. See on AppTextInput, the onChangeText prop sets the todoText value, and the todoText is the value of the input itself (this is called two way binding on the todoText state variable).
Next we use the useDispatch hook from react-redux. This hook returns a reference to the dispatch function from the redux store. We can use this to dispatch any action as needed.
Our handleSubmit function abstracts the logic for submitting a todo. If there is no text, we do nothing, otherwise we use our dispatch reference to dispatch an addTodo action creator with our todo text. Finally we set the local todoText state variable to an empty string in order to clear the input box.
We use the handleSubmit function on press of the submit button, as well as onSubmitEditing of the text input (this handles adding a todo when return is pressed on the device keyboard).

That’s it! There aren’t many extra pieces to add in order to create a stateful react component with redux. I’ll now let you build the other 3 components. Don’t forget to use the useSelector hook with your todo selectors! The hook will return whatever your selector does.

TodosList

I should mention that the final version of my project includes todo filters, which haven’t been spoken about here. These filters largely affect this component, so here is the code minus the filter logic.

import {FlatList, StyleSheet} from 'react-native';
import {shallowEqual, useSelector} from 'react-redux';

import React from 'react';
import TodoItem from './TodoItem';
import {selectTodoIds} from '../store/todos/selectors';

const TodosList: React.FC = () => {
  const todoIds = useSelector(selectTodoIds, shallowEqual);

  return (
    <FlatList
      data={todoIds}
      keyExtractor={(id) => id.toString()}
      renderItem={({item}) => <TodoItem id={item} />}
      style={styles.container}
    />
  );
};

const styles = StyleSheet.create({
  container: {
    width: '100%',
  },
});

export default TodosList;

/src/components/TodosList.tsx

There is an optimisation in this component worth talking about. You might ask why we pass a list of ids instead of a list of todos. The reason for this is so that we can use the shallowEqual function as a second argument to our useSelector hook.

The second parameter of the useSelector hook is a comparator. This comparator is used to compare items from a new state, to items from the previous state. If the item has not changed, then redux will not trigger a re-render of this item.

If we did not use this comparator, then every time we update a todo’s ‘completed’ status, the whole list of todos would re-render!

You can also look at using the [useMemo](https://reactjs.org/docs/hooks-reference.html#usememo) hook as an alternative to using a comparator.

Wrapping Up

In this guide, I haven’t included the addition of the todo status filters (all, complete, incomplete), but you should now have the tools to build this yourself.

The steps you need to take for this are:

Create a /src/store/filters directory.
Create the necessary types, actions, reducer and selectors.

(Hint: you'll need a new todos selector that selects todos based on the current filter!)
Create the FilterButton component (Hint: it should dispatch a CHANGE_FILTER action creator!).
Add some FilterButton components to either the TodosScreen or TodosList component.

The source code on my github repository has this completely implemented if you need help or inspiration

⭐️ Thanks for reading! ⭐️

There are many, many more pieces to learn about React Native, Redux and application architecture, like: testing, routing, native features, making api calls, asynchronous state updates (Redux Saga/Thunk).

Resources

React Tutorial - useful for learning React!
React Hooks - if you don’t know what react hooks are or how to use them, the official docs are great!
React Native Basics - useful for learning React Native!
Redux Essentials, Redux - Getting Started and Redux with React - all on the redux official docs, great source of information for redux newbies!
Typescript in 5 mins! - learn the basics of Typescript!

Using Native Incoming Call UI for your React Native App (How I Wrote My First Native Module)

Wed, 17 Mar 2021 00:00:00 GMT

A story about my struggles implementing native call display functionality and how this led me to writting my own native module for android. Suitable for developers who are interested in how to get started with native modules for android within React Native or implementing native UI for incoming calls.

TL;DR

A project I was working on required native call UI to be implemented so that when users received a call notification from firebase their phone would ring as if they were receiving a native call. This was a really interesting feature to build but it did not come without difficulty. The library react-native-callkeep was used to implement this solution, this library acts as a fully managed connection service for implementing call UI using react native. However, the project had its own in-call UI we wanted to use, therefore we only wanted to use the incoming call functionality of callkeep. This solution seemed to work fine for iOS with the base recommended implementation, with the downside that it didn't work as smoothly for the latest android versions we wanted to run (android with a minimum api of 27). To get android working properly for these versions I had to write a native module which mimicked the backToForeground function from react-native-callkeep.

Native call UI trying to be implemented:

Intro

When I first realised the project I was working on would need an incoming call display that transitions into a React Native app I (foolishly) didn't think much of it, and assumed a simple library would solve most of my worries. As a team we decided to use the react-native-callkeep library which seemed to have all of the functionality required, which was to be able to display incoming call UI after a notification is received.

This implementation however, started to show a few issues when testing functionality on the latest android versions. These issues then led me down a track which ended with implementing a native android module. Writing this native module eventually did result in the desired functionality, although, to properly explain my decision into doing this I need to start from the beginning.

Using react-native-callkeep

The base implementation of callkeep is pretty straight forward and works well out of the box, when initially experimenting with it, the only problem I ran into was that it wouldn't display the native incoming call UI on certain android emulators. Switching from an android 11 emulator to an android 10Q emulator seemed to do the trick (note: this was only a problem for the emulator as the same functionality seemed to work on a real android 11 phone).

After following the basic installation guide (Found here), I would recommend setting up a callkeep class in your project. The basics to get the display incoming call UI to work for my project was:

import RNCallKeep, { AnswerCallPayload } from 'react-native-callkeep';

type DidDisplayIncomingCallArgs = {
  error: string;
  callUUID: string;
  handle: string;
  localizedCallerName: string;
  hasVideo: string;
  fromPushKit: string;
  payload: string;
};

export class CallKeep {
  private static instance: CallKeep;
  private callId: string;
  private callerName: string;
  private callerId: string;
  private isAudioCall: string | undefined;
  private deviceOS: string;
  private endCallCallback: Function | undefined;
  private muteCallback: Function | undefined;
  public IsRinging: boolean = false;

  constructor(
    callId: string,
    callerName: string,
    callerId: string,
    deviceOS: string,
    isAudioCall?: string
  ) {
    this.callId = callId;
    this.callerName = callerName;
    this.callerId = callerId;
    this.isAudioCall = isAudioCall;
    this.deviceOS = deviceOS;

    CallKeep.instance = this;
    this.setupEventListeners();
  }

  public static getInstance(): CallKeep {
    return CallKeep.instance;
  }

  endCall = () => {

    RNCallKeep.endCall(this.callId);

    if (this.endCallCallback) {
      this.endCallCallback();
    }

    this.removeEventListeners();
  };

  displayCallAndroid = () => {
    this.IsRinging = true;
    RNCallKeep.displayIncomingCall(
      this.callId,
      this.callerName,
      this.callerName,
      'generic'
    );
    setTimeout(() => {
      if (this.IsRinging) {
        this.IsRinging = false;
        // 6 = MissedCall
        // https://github.com/react-native-webrtc/react-native-callkeep#constants
        RNCallKeep.reportEndCallWithUUID(this.callId, 6);
      }
    }, 15000);
  };

  answerCall = ({ callUUID }: AnswerCallPayload) => {

    this.IsRinging = false;

    navigate('somewhere'); // navigated to call screen in our app
  };

  didDisplayIncomingCall = (args: DidDisplayIncomingCallArgs) => {
    if (args.error) {
      logError({
        message: `Callkeep didDisplayIncomingCall error: ${args.error}`,
      });
    }

    this.IsRinging = true;
    RNCallKeep.updateDisplay(
      this.callId,
      `${this.callerName}`,
      this.callerId
    );

    setTimeout(() => {
      if (this.IsRinging) {
        this.IsRinging = false;
        // 6 = MissedCall
        // https://github.com/react-native-webrtc/react-native-callkeep#constants
        RNCallKeep.reportEndCallWithUUID(this.callId, 6);
      }
    }, 15000);
  };

  private setupEventListeners() {
    RNCallKeep.addEventListener('endCall', this.endCall);
    RNCallKeep.addEventListener(
      'didDisplayIncomingCall',
      this.didDisplayIncomingCall
    );

  private removeEventListeners() {
    RNCallKeep.removeEventListener('endCall');
    RNCallKeep.removeEventListener('didDisplayIncomingCall');
    this.endCallCallback = undefined;
  }
}

This setup will get you the basics that you will need to handle displaying the incoming call UI along with a base setup for answering and ending calls. The key functions here to get only the incoming call UI working is the RNCallKeep.displayIncomingCall and RNCallKeep.endCall.

Initial problems with android

After my initial experiments with callkeep it seemed that IOS worked fine out of the box, however, there were some difficulties to get the same functionality working for android. The difficulties I came across with android are summarised below:

Emulator was not displaying the incoming call UI when correct command was called (RNCallKeep.displayIncomingCall)
- After experimenting with different emulators and getting the functionality to work, this led me to the conclusion that the android 11 emulator was not set up correctly to use this library.
- Changing to an android 10Q api 29 android fixed this issue, and the incoming call UI was successfully displayed
Answering the call from the incoming UI display on android was not navigating me to the app, the UI acted as if I had never answered the call.
- Solving this issue required some experimenting, but in the end was solved by calling RNCallKeep.endCall on the callAnswered listener and then navigating to the correct page on our app.
- Calling the RNCallKeep.endCall seemed to prompt android to stop displaying the UI and run the rest of the commands in the answered call listener function
- This step needed to be done on android only.
When a call was received, if android was locked, the user would not be navigated to the app but would instead be stuck at their home screen.
- The functionality required here instead of this issue would be for the user to be prompted to unlock their phone, and then be navigated to the app.
- This condition was a bit trickier and required me to dive into the realms of native app development as discussed below.

Writing a native module for android

Fulfilling the functionality required by the lockscreen took some further investigation, the callkeep library instructs you to call the android only function RNCallKeep.backToForeground to bring the react-native app to the foreground. This seemed to work while the app was in background mode but not while the app was in a lockscreen state. Looking into this function I found that it was using some java functions that were deprecated for the android versions necessary for our project. This led me to recreate this backToForeground function natively with an updated set of java functions that would work for the required versions.

Since I am working in React Native to write native code and call it from typescript/javascript I needed to use the React Native bridge. The android bridging documentation from React Native (Found here) has all of the necessary steps required to set up the java module and call it from within your app. Below is an example of the java module I created to start the app activity and request the user to unlock their phone.

package com.ExampleProject;

import android.app.Activity;
import android.app.ActivityManager;
import android.app.KeyguardManager;
import android.content.Context;
import android.content.Intent;
import android.net.Uri;
import android.os.Build;
import android.os.Bundle;
import android.util.Log;
import android.view.WindowManager;

import androidx.annotation.RequiresApi;

import com.facebook.react.bridge.Arguments;
import com.facebook.react.bridge.NativeModule;
import com.facebook.react.bridge.Promise;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.bridge.ReactContext;
import com.facebook.react.bridge.ReactContextBaseJavaModule;
import com.facebook.react.bridge.ReactMethod;
import com.facebook.react.bridge.ReadableMap;
import com.facebook.react.modules.core.DeviceEventManagerModule;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class CallkeepHelperModule extends ReactContextBaseJavaModule {

  private static ReactApplicationContext reactContext;

  CallkeepHelperModule(ReactApplicationContext context) {
    super(context);
    reactContext = context;
  }

  @Override
  public String getName() {
    return "CallkeepHelperModule";
  }

  @RequiresApi(api = Build.VERSION_CODES.O_MR1)
  @ReactMethod
  public void dismissKeyguard(Activity activity){
    KeyguardManager keyguardManager = (KeyguardManager) reactContext.getSystemService(
      Context.KEYGUARD_SERVICE
    );
    boolean isLocked = keyguardManager.isKeyguardLocked();
    if (isLocked) {
      Log.d("CallkeepHelperModule", "lockscreen");
      keyguardManager.requestDismissKeyguard(
        activity,
        new KeyguardManager.KeyguardDismissCallback() {
          @Override
          public void onDismissError() {
            Log.d("CallkeepHelperModule", "onDismissError");
          }

          @Override
          public void onDismissSucceeded() {
            Log.d("CallkeepHelperModule", "onDismissSucceeded");
          }

          @Override
          public void onDismissCancelled() {
            Log.d("CallkeepHelperModule", "onDismissCancelled");
          }
        }
      );
    } else {
      Log.d("CallkeepHelperModule", "unlocked");
    }
  }

  @ReactMethod
  public void startActivity() {
    Log.d("CallkeepHelperModule", "start activity");
    Context context = getAppContext();
    String packageName = context.getApplicationContext().getPackageName();
    Intent focusIntent = context.getPackageManager().getLaunchIntentForPackage(packageName).cloneFilter();
    Activity activity = getCurrentActivity();
    boolean isRunning = activity != null;

    if(isRunning){
      Log.d("CallkeepHelperModule", "activity is running");
      focusIntent.addFlags(Intent.FLAG_ACTIVITY_REORDER_TO_FRONT);
      activity.startActivity(focusIntent);
      dismissKeyguard(activity);
    } else {
      Log.d("CallkeepHelperModule", "activity is not running, starting activity");
      focusIntent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK + Intent.FLAG_ACTIVITY_REORDER_TO_FRONT);
      context.startActivity(focusIntent);
    }
  }

  private Context getAppContext() {
    return this.reactContext.getApplicationContext();
  }
}

The main function of this module is startActivity, which when called gets the context for the React app, if the app is running the function brings the app to the foreground and checks if the keyguard needs to be dismissed, otherwise if the app is not running this function starts the apps activity as a new task. The keyguard is then dismissed by calling the dismissKeyguard function which uses the KeyguardManager to check if the android is locked or unlocked, and asks the user to dismiss the keyguard if the phone is locked.

Putting all of these steps together I could then call the native module in my react app whenever a call was accepted from the incoming UI using the answerCall listener function like so:

import { NativeModules } from "react-native";
import RNCallKeep from "react-native-callkeep";

answerCall = ({ callUUID }: AnswerCallPayload) => {
  if (this.deviceOS === "android") {
    const { CallkeepHelperModule } = NativeModules;
    CallkeepHelperModule.startActivity();
    RNCallKeep.endCall(this.callId);
  }
  this.IsRinging = false;
  navigate("somewhere");
};

Conclusion

Callkeep was able to be implemented as an incoming UI solution only, in both IOS and android platforms. On android, native code was written so that answering the app from the lockscreen would start the react native app and navigate you to the correct screen. After these adjustments the React Native callkeep library ended up being a great fit for this project, performing all the required functionality. I hope this article is able to help anyone deciding between solutions to display incoming call UI, and hopefully cut down on some research and debugging time. I will finally leave you with a gif of the working functionality in all its glory, happy coding!

How to easily handle admin authentication and permissions in Django with Okta

Tue, 09 Mar 2021 00:00:00 GMT

In my project, we are developing an admin interface to create and share documents to users who can see them and download them from the website we are creating.

However, the administration platform is dedicated to multiple teams from various countries. We don’t want them to be able to see or interact with each other's documents. Besides, those teams are meant to change, and they should manage themselves. For each team, we want to create a super admin account able to create, delete and change permissions for its team members only.

I will show how we handled this challenge with an effortless and fast solution.

TL;DR:

In this article, I will developp three main points:

Why use Okta to manage users and permissions, especially in Django admin?
How to add single sign-on connection using the mozilla-django-oidc library?
How to manage user permissions in Django from Okta ?

Why use Okta with Django admin?

One of Django's valuable features is the management of permissions for various users. Though, it is not always simple to separate users into distinct independent groups and to allow some of them to manage only a fraction of it without seeing the other users. Besides, we wanted a system allowing the admins to connect on the Django platform with their admin account.

For our application, the authentication and user management system must:

be secure (standard auth flows, follows security best practices, possible to implement two-factor authentication)
be fast and scalable.
be easy to work with from a developer standpoint.
be simple enough for the team managers.

To handle those requirements, we decided to use Okta tools. Okta is a company founded in 2009 providing a cloud software. This software allows you to manage and secure user identity and access management. Okta’s CEO announced they had over 100 million registered users in 2019. Besides, Okta is top-notch on up-to-date security measures and auth flows.

Okta main functionalities

As seen in the graph below, Okta offers four major functionalities:

A fully automated onboarding sending emails to users to help them create their account.
Identity access and management to provide group management and two-factor authentication.
Accesses are given in real-time when adding users and changing groups.
Okta cares about security and has a protocol in place in case of an incident.

In Okta, it is simple for an admin with the correct rights to create other users and grant them the right groups. This way, at each level of the organization, users and their permissions are manageable effortlessly.

Example of a user created in Okta with its groups

I will now explain how we integrated Okta with Django admin and how we used it to manage our users’ permissions.

Implement authentication workflow to Django through Okta

To add the admin connection with Okta to Django admin, our team used mozilla-django-oidc library. “A lightweight authentication and access management library for integration with OpenID Connect enabled authentication services.” OpenId Connect, also known as OIDC, is an identity layer on top of the OAuth 2.0 protocol. The End-User identity can be verified with the authentication thanks to an Authorization server. OpenID Connect introduces the concept of an id_token, which is a security token allowing the client to verify the user's identity. The id_token also gets basic profile information about the user. It introduces the UserInfo endpoint, an API returning information about the user.

The Sign-in protocol used follows the steps described in the next diagram.

Sign-in protocol diagram

We configured the callback url with the client_id and added the client_secret in our settings to acquire the token from Okta as written in the library documentation. In our case, we used a RS256 algorithm.

# In settings.py

# Okta
OKTA_DOMAIN = os.environ.get("OKTA_DOMAIN")
OKTA_TOKEN = os.environ.get("OKTA_TOKEN")

# User information
USER_CRM_ID = None
USER_EMAIL = None

# Okta Admin
OKTA_ADMIN_DOMAIN = OKTA_DOMAIN
OIDC_RP_SIGN_ALGO = "RS256"
OIDC_OP_JWKS_ENDPOINT = f"{OKTA_ADMIN_DOMAIN}/oauth2/v1/keys"
OIDC_OP_AUTHORIZATION_ENDPOINT = f"{OKTA_ADMIN_DOMAIN}/oauth2/v1/authorize"
OIDC_OP_TOKEN_ENDPOINT = f"{OKTA_ADMIN_DOMAIN}/oauth2/v1/token"
OIDC_OP_USER_ENDPOINT = f"{OKTA_ADMIN_DOMAIN}/oauth2/v1/userinfo"
OIDC_RP_SCOPES = "openid profile email groups"
OIDC_RP_CLIENT_ID = os.environ.get("OKTA_ADMIN_CLIENT_ID")
OIDC_RP_CLIENT_SECRET = os.environ.get("OKTA_ADMIN_CLIENT_SECRET")
OIDC_VERIFY_SSL = True
LOGIN_REDIRECT_URL = f"{BASE_URL}/admin/"
OIDC_REDIRECT_URL = f"{BASE_URL}/admin/oidc/callback/"
OIDC_AUTH_REQUEST_EXTRA_PARAMS = {"redirect_uri": OIDC_REDIRECT_URL}

It is now possible to override the login template. We added a button allowing users to connect with their Okta account.

# In templates/admin/login.html
# Admins can also log in through the usual credentials flow
{% extends "admin/login.html" %}

{% block extrastyle %}
   {{ block.super }}
   <style>
       #content-main {
           float: none !important;
       }

       .okta_login_container {
           margin-top: 15px;
           display: flex;
           justify-content: center;
       }
   </style>
{% endblock %}

{% block content %}
   {{ block.super }}

   <div class="okta_login_container">
       <a href="{% url 'oidc_authentication_init' %}">Or Login With Okta</a>
   </div>
{% endblock %}

New button to log in with Okta

After adding a way to connect through Okta to Django Admin, we must save and update users. We are getting their information from Okta and adding them in Django. To do so, we override the class OIDCAuthenticationBackend from mozilla-django-oidc. We can get the information about the user's verified email and create or update it in Django, without any permissions for the moment.

class OktaAdminAuthenticationBackend(OIDCAuthenticationBackend):
   def verify_claims(self, claims: OktaClaim) -> bool:
       return (
           super().verify_claims(claims)
           and claims.get("email_verified", False)
       )

   @transaction.atomic
   def create_user(self, claims: OktaClaim) -> User:
       user: User = self.UserModel.objects.create_user(
           claims.get("email"),
           None,  # password
           first_name=claims["given_name"],
           last_name=claims["family_name"],
       )
       user.save()

       return user

   @transaction.atomic
   def update_user(self, user: User, claims: OktaClaim) -> User:
       """Update existing user with new claims, if necessary save, and return user"""
       user.first_name = claims["given_name"]
       user.last_name = claims["family_name"]
       user.save()

       return user

Administrators can log in to the application with their Okta credentials. I will show you how to add their permissions synchronizing it with Okta.

Add groups and permissions from Okta in Django Admin

After adding users to Django admin, we must grant them the right permissions to manage the customer’s website. In my project, we had two types of administrators for each team:

Admins with basics rights. They can create documents for their team’s users but can’t publish them.

We named them Admin - [Group Name] in Okta.
Controllers. They are admins who can validate and publish documents for their team’s users.

We named them Admin - [Group Name] Controller in Okta.

Example of two groups in Okta

In Django Admin, we decided to create two types of group. The first one is a group for each team with basic permission. The other is a general group called Administrator Controller with more permissions to validate and publish documents. The graph below shows the flow to attribute groups to administrators.

User and group creation flow

To handle those permissions, we get user information from the Okta response which contains the user groups. We can separate those groups according to our business rules as described above.

def handle_controller_groups(self, groups):
    groups_with_controller = set()
    for group in groups:
        # remove Controllers groups and add generic 'Administrator Controller'
        if bool(re.search("[Cc]ontroller", group)):
            groups_with_controller.add(ADMINISTRATOR_CONTROLLER)
        else:
            groups_with_controller.add(group)
    return groups_with_controller

def get_user_groups(self, claims: OktaClaim) -> QuerySet:
    claim_groups = claims.get("groups")
    assert claim_groups
    groups = [group[8:] for group in claim_groups]  # remove `Admin - ` prefix

    groups_with_controllers = self.handle_controller_groups(groups)

    return Group.objects.filter(name__in=groups_with_controllers)

Finally, we use those functions to get users permissions when we create and update them.

def verify_claims(self, claims: OktaClaim) -> bool:
    return (
        super().verify_claims(claims)
        and claims.get("email_verified", False)
        and claims.get("groups") is not None
    )

@transaction.atomic
def create_user(self, claims: OktaClaim) -> User:
    user: User = self.UserModel.objects.create_user(
        claims.get("email"),
        None,  # password
        first_name=claims["given_name"],
        last_name=claims["family_name"],
    )
    user.groups.set(self.get_user_groups(claims))
    user.save()
    return user

@transaction.atomic
def update_user(self, user: User, claims: OktaClaim) -> User:
    """Update existing user with new claims, if necessary save, and return user"""
    user.first_name = claims["given_name"]
    user.last_name = claims["family_name"]
    user.groups.set(self.get_user_groups(claims))
    user.save()

    return user

Our administrators can create their documents with the right permissions and only their team members can see them. Besides, controller admins can manage their team and review documents to publish them.

Conclusion

With this implementation, administrators don’t need to call us to manage their teams and they only need their Okta credentials to do their various jobs.

One of the limitations is that if a new group is created, it needs to be added in Okta and Django to allow users to have their permissions when they log in. We decided we didn’t want to create groups when we didn’t recognize them in Django to avoid issues with permissions as administrators can upload sensitive information.

I hope you enjoyed reading this article and you learned a new way to handle administrators permissions with Okta and Django admin. Hope you’ll give it a try!

You can also contact one of our python expert:

Useful links

How FlexBox has (not so much) changed my life

Fri, 05 Mar 2021 00:00:00 GMT

When times are hard, I sometimes settle down and think back to my recent debut on the web. Nostalgic, I think back to the very first web course I attended. A course on HTML code, an introduction to CSS, and a presentation of Bootstrap, that's all. What marks me with hindsight, it's not so much the fact that it's quite sad to discover the web this way given the richness and diversity brought by all the frameworks that already existed at that time. Without talking about frameworks, even a few lines of Javascript could have made it much more interesting. No, what marks me are those hours spent trying to understand CSS positioning.

"Why is an element with an absolute position positioned relative to its closest ancestor having a relative position? How can an element have a relative position if I don't specify what it is relative to". It's all those hours spent looking for THE property that will magically give the rendering I want without destroying the rest of the style of the page. Those are the moments when proud to finally have the visual I was looking for, I decided to share the code with friends and finally realized that a slight difference in screen resolution sent me back to square one.

This until a new web course, one year later. HTML, CSS, a bit of javascript, nothing transcendental. First lab session, I was asked to place frogs on water lilies using Flexbox. "What's that joke?" would you say? Well, that's when FlexBox was about to change my life.

Disclaimer

The purpose of this article is in no way to sell the FlexBox framework or to promote it ,since it is universally known. Its only purpose is to share my experience and my personal misadventures related to the CSS positioning and the use of this framework, all in a humorous and playful tone. On that note, enjoy your reading!

History of the css positioning

CSS Flexible Box Layout Module (or FlexBox) appeared with the arrival of the CSS3 module at the end of the 2010s. At the same time CSS Grid Layout, also very widespread, appeared. But I will not discuss it here. The emergence of these layouts was motivated by the popularization of "liquid layouts" (which adapt to the page and reorganize themselves dynamically) as well as responsive interfaces to adapt to the diversification of media sizes (with the increase in screen sizes and the emergence of smartphones). Previously, developers had to rely on javascript frameworks dedicated to the design of interfaces such as Bootstrap, or manage with existing CSS properties (relative sizing using % units, media queries, ...). "Vanilla" CSS positioning is based on 3 main properties: display, float, and position. Anyone who has already made a web interface has already played with these three properties to get the desired UI. Out of nostalgia and for science, I decided to take an interface made with FlexBox and try to reproduce it as faithfully as possible using these properties.

Comparison

Although React appeared after FlexBox, I allow myself to use React and styled-components for reasons of comfort and simplicity. So I took a whole page from a project I recently worked on and challenged myself to reproduce it as faithfully as possible using the native CSS positioning properties. So I'm going to base my comparison on several criteria: appearance, number of lines, responsiveness, and uniformity across browsers. I will also be interested in more subjective characteristics such as code clarity or complexity.

Aspect

First we will compare the appearance of the two pages.

Target (With FlexBox)

Result (Without FlexBox)

From the point of view of appearance, we can see that the result is quite close to the objective. The only noticeable difference is the spacing between the elements on the right-hand side which is not the same on the two images.

On this point, we can say that it is possible to manage without FlexBox. But at what cost?

Code length

We will now compare the length of the code in both cases. First of all, on a simple element like the header, the CSS code looks like this:

With FlexBox

            import styled from 'styled-components'
            export default {
                Header: styled.div`
                    width: 100%;
                    height: 160px;
                    padding: 25px 15px;
                    background-color: palegoldenrod;
                    display: flex;
                    flex-direction: column;
                    justify-content: space-between;
                    align-items: flex-start;
                `,
                TitleContainer: styled.div`
                    display: flex;
                    flex-direction: row;
                    align-items: center;
                `,
                Pictogram: styled.div`
                    background: url('https://picsum.photos/id/475/400/350') no-repeat;
                    background-size: cover;
                    width: 40px;
                    height: 35px;
                `,
                Title: styled.span`
                    font-weight: bold;
                    font-size: 40px;
                    line-height: 40px;
                    margin-left: 20px;
                `,
                SearchBar: styled.input`
                    background-color white;
                    width: 100%;
                    padding: 10px;
                `,
                }

Without FlexBox

            import styled from 'styled-components'
            export default {
                Header: styled.div`
                    width: 100%;
                    height: 160px;
                    padding: 25px 15px;
                    background-color: palegoldenrod;
                    position: relative;
                `,
                TitleContainer: styled.div`
                    position: relative;
                `,
                Pictogram: styled.div`
                    background: url('https://picsum.photos/id/475/400/350') no-repeat;
                    background-size: cover;
                    width: 40px;
                    height: 35px;
                    display: inline-block;
                    position: absolute;
                    top: 50%;
                    transform: translateY(-50%);
                `,
                Title: styled.span`
                    font-weight: bold;
                    font-size: 40px;
                    line-height: 40px;
                    margin-left: 60px;
                `,
                SearchBar: styled.input`
                    background-color: white;
                    width: 100%;
                    padding: 10px;
                    position: absolute;
                    bottom: 0;
                `,
                SearchBarWrapper: styled.div`
                    position: relative;
                    height: 70px;
                `,
                }

We can see that the code without FlexBox has 3 more CSS properties (25 against 22) and one more element (a wrapper around the SearchBar).

On the whole page, the code without flexbox has 2 more lines (108 against 106) and 1 more element (17 against 16). Finally, here again, CSS code without FlexBox is not longer than CSS code with FlexBox.

Responsiveness

However, as I stated earlier, one of the reasons for the existence of FlexBox is its "liquid" nature. It is a layout that came into being with the diversification of responsive interfaces. It is therefore interesting to see if our example can highlight this property. To do so, we will observe the rendering of our interface for different screen sizes and ratios.

With FlexBox

Large Laptop

Laptop

Tablet

Large Mobile

Mobile

Without FlexBox

Large Laptop

Laptop

Tablet

Large Mobile

Mobile

Surprisingly enough, we notice that our interface behaves well (or even better) without FlexBox when we reduce the screen size. However, problems that occur on a too-small screen with FlexBox are easily adjustable. We can therefore note that for simple, rather geometrical interfaces like ours, the UI behaves relatively well when the screen size and ratio changes.

Uniformity

Another criterion that might be interesting to look at is uniformity across browsers. Indeed, the behavior of a CSS code can be quite different on Chrome, Internet Explorer and Edge. So I decided to observe the rendering of the same interface on these three browsers.

With FlexBox

IE11

Edge88

Chrome

Without FlexBox

IE11

Edge88

Chrome

Once is not customary, against all my expectations, there is no difference between the behavior of the style on different browsers. I even have my doubts. Flexbox would just be a masquerade and would have no advantage? Have I become an expert in pure CSS? Joking aside, the fact is that my user interface here is quite simple, quite structured, square. So the power of flexbox is not being expressed at its true value. I don't think I've chosen the most suitable example.

Personal feeling

In the end, in this experience, the only notable difference I was able to raise is quite subjective and concerns my personal feeling about the clarity and complexity of the code. Although it is difficult to quantify these criteria, I felt a real difference when coding.

Code complexity

As far as the complexity of the code is concerned, if I refer to the time I spent on it, the handling of the two methods is quite different. The code with flexbox is quite intuitive. Once the necessary time has been spent getting familiar with the tools, the development flow is fairly fluid and fast and requires very little adjustment. Conversely, without FlexBox, the properties are basic CSS properties but I find the development flow much less reliable. The code requires a lot of adjustments, especially when it comes to making the interface stable on different media and different browsers. I feel a bit like cooking or doing magic.

Code clarity

In terms of clarity, it's again subjective but the code is much clearer in my opinion by using flexbox. The property names are self-explanatory and it's easy to understand the interface structure or change an alignment at a glance. Moreover, I have experienced this by taking code I wrote in my early school days (without FlexBox) and trying to understand it and modify it a bit. Apart from the lack of rigor in the writing and the structure of the code in general, I had a lot of trouble making slight modifications to the alignment or the ordering of my interface. On the other hand, I took more recent code (with FlexBox) and I had much less difficulty reworking it.

Review

Finally, even if my experience doesn't show it so much, FlexBox is known to have revolutionized liquid layouts and responsive interfaces. It has replaced the overuse of media queries, relative units (%, em, ...), or javascript to dynamically change properties. As a reminder, I don't consider myself an expert in CSS (far from it). There are most likely better ways to do things without using FlexBox. This article has no scientific value and is only intended to report on my personal experience.

My aim was mainly to show you how FlexBox has (not so much) changed my life.

The ultimate irony: For someone writing an article on CSS positioning, if only you knew how much time I spent to position my images for comparison...

Why Aws Step Functions Is Not the Best Tool for Business Processes

Wed, 27 Jan 2021 00:00:00 GMT

I recently worked on a project where we needed to orchestrate a process, applying for loan to be precise, which could take weeks. To do this, we decided on using a serverless architecture and specifically AWS Step functions. The use of such a framework helped us immensely to organize the flow of the process but it wasn't without a few bumps along the road. My aim in this article is to warn you about those bumps and help you have a smoother ride.

I. AWS stacks and their limits

1. Cloud-formation stack

Cloudformation is a service from AWS which allows you to create and manage linked AWS resources (which are themselves other services). All the resources that we used in our project were initially in a single CloudFormation stack. Let's look at an example of a stack :

Here we can find our Lambdas, as well as Logs and if we scrolled further we could also find the state-machines we used. In terms of code, a stack corresponds to a configuration file called serverless.yml. In this file all the resources , environment variables, and outputs, are defined.

2. Cloud formation stacks limits

A stack has multiple limits which can quickly be reached. You can find the complete list in the AWS docs but we will only talk about the one encountered in the project.

A stack can contain a maximum of 200 resources. 200 state machines and Lambdas would indeed be a lot but there are other resources we have to take into account. Indeed, for each Lambda we create we also create a Log group resource and a Version resource. State Machines also have a Role resource. There are other resources we use like SQS, S3 buckets, DynamoDB tables. As mentioned above we had all our resources in a single stack. As you can already guess we reached that limit of 200 resources. We were unable to develop any new features until we found a solution to this limit. As such, we were hard-pressed to find a solution.

The first thing we thought of was, is this really a limit? Could our dev ops team set this limit to higher value or could we even contact the AWS support to ask for a less restrictive limit? We tried but to no avail. This limit is one you will have to live with and can't cheat your way around. Ok, so the quick fix we hoped for is a bust, let's get thinking about real solutions.

We knew we could split the stack into multiple ones but that would require a lot of refactoring and we already saw glimpses of the problems it would generate. So we searched for a smarter approach and we found a plugin that did exactly what we wanted: serverless-plugin-split-stacks, or so we thought. Three of us tried getting the plugin to work like we wanted without success. But what really made us give up on this solution was that we realised more and more that the real problem was the faulty architecture of our serverless application. Even if we could fix the resource problem with the plugin (if we got it to work) other problems might arise from the fact that we bunched up everything into one stack. So ultimately we decided that the pragmatical solution was to split our stack into multiple sub stacks.

3.Difficulties with concurrent stack updates

I'll spare you the details of how we split the stack, but we essentially had different services which we put in separate stacks and kept one main stack with nothing more than the orchestration between the different services. You can see in the image below an example of the split works. What I want to talk about here is the new problem brought on by the split.

For the split to work we have to export the resources we intend to use in the main stack, namely the different lambdas and sub state-machines. AWS allows us to do this quite easily by defining exports with region specific names that we can then import in another stack. The problem comes when we want to update an exported resource. Before going into more detail you will need to understand how AWS updates a resource. There are two types of updates :

Updating only the core of the resource, for example changing the code of a lambda
Updating the configuration of the resource, for example the name of the resource

You can find out more about Cloud Formation resource updates here. In the first case, AWS can update the resource without changing anything but the changes we asked for. In the second case AWS has to replace the resource with a new one. In order to do so, it has to delete the resource and create a new one.

Now we can come back to our problem. What happens when an exported resource needs to be removed to update it? Well AWS won't allow you to deploy your stack because you will be trying to remove a resource currently used by another stack. Quite the pickle. The only way to go around this is to duplicate the resource, apply the update on the new copy, change the import to use the new copy and finally remove the old copy. We realised this problem existed before we went ahead with splitting the stacks. We wagered that those updates would seldom be needed. I can confirm that a few months later those updates were very rare, and the proposed workaround worked, so all in all it was a good call. I choose to talk about this problem because it brings me to a larger underlying problem: handling versions in an AWS serverless framework.

II. Problems caused by differences in versions between state machines and lambdas

1.UseExactVersion

The problem we encountered with the limits of the stack brought us to another problem : handling changes in version. For me this problem is at the core of what I want to talk about in this article. We knew of it before but until this point it wasn’t too problematic. Indeed AWS gives us a tool to handle changes in version. For example : what happens when you deploy a new state machine with one lambda updated? Do the old state machines use the new lambda when they get to the step in question?

To answer these questions AWS gives us the very useful useExactVersion. By setting this parameter to true in the configuration of a step we tell AWS to synchronise the version of the state machine with the lambda called in the step. This way AWS keeps a version of lambda for each version of currently running state machines. To answer our initial question : no, AWS will not call the new lambda for old state machines. So, as I said earlier, with this tool we hadn’t really encountered big problems linked to versioning. But of course just like the stack had its limits so does useExactVersion.

2. Secondary state-machines

UseExactVersion works for lambdas called in a step. What happens when a step calls another state machine? In that case useExactVersion does not apply. Say we update state machine B which is called by state machine A. An old state machine A in version 1 will call a state machine B in the new version 2. If you have breaking changes between the versions, then you will have a problem.

For example, we wanted to sanitize the inputs of our whole process. To do so, we need to change the inputs of all our steps. For the steps calling lambdas there was no problem thanks to useExactVersion but the whole process still crashed because old sub state machines were expecting the complete input instead of the sanitized version. The only way we could proceed was to duplicate the whole state machine like we did when we wanted to split the stacks. This would lead to a third concurrent stack, which we judged too costly and lead us to abandon the idea of sanitizing the inputs.

3. Modifying the steps

That is not the only problem we encountered with versioning. Adding a new step to an existing state machine is not a problem and modifying the execution of a step is also feasible. But what happens when you want to delete a step? We thought about it and concluded that if we kept the code for the lambda that was called, it wouldn’t be a problem. Indeed, already running state machines would still have the definition of the step and since the code for the called lambda still exists, everything would be fine. Of course it wasn’t.

To be able to execute certain tasks, to call lambdas for example, AWS uses roles. For a step to be able to call a specific lambda function it needs the associated role. What we did not anticipate was that when we removed the step from the configuration of the state machine, the role generated for that step was also removed. That means that when an old state machine tried to call the removed step it did not have the role necessary. We had two options:

Replace the automatic generation of roles and manually add the needed roles in the configuration (over a hundred roles)
Keep the step in the state machine and put it somewhere it would not be reached. Although the second option felt like hiding the mess under the rug, we were pressed for time, so we decided it was the more pragmatic solution.

III. Conclusion

With these specific examples to illustrate the difficulties of versioning with AWS Cloudformation and Step Functions, I want to more generally show you that state machines are made to be deployed rarely. If you want to deploy a new version it is best to deploy a whole new state machine rather than update the current one. Now that is quite a limiting aspect, since we often work with products that are already in production, and we continuously upgrade them. If you need to use a state machine like architecture and can't afford to seldom deploy new versions I do have a recommendation. In a new project we have been using Camunda which allows you to code state machines on your Springboot (a Java framework) server. I found that it has many advantages over AWS. Mostly it has three functionalities I like :

It is less complex than AWS (no roles for example) and has a gui to create the state machine
It allows you to migrate state machines from one version to another if there are no breaking changes
It has an integrated REST engine that allows you to easily manipulate the state machine (change the current step, stop/start state machines etc.)

With that said, I hope this article either helped you on your journey with AWS and State machines or better prepared you to better start using these technologies.

Managing multiple loading states in Angular

Fri, 22 Jan 2021 00:00:00 GMT

When developing an Angular application, you will probably have to perform HTTP requests. Whether it be fetching data from a separate backend application or an external api, your application might have to wait for a response at some point.

Properly handling the state of an HTTP request in a web application is a common use case. In a large codebase, it quickly becomes a critical matter of architecture. This matter is even more important than Angular is a verbose framework.

In this context, your solution to this concern has to be :

Simple enough to keep your codebase readable
Reusable enough to keep your codebase sane and to save time when creating new pages or components
Adaptive enough to handle the different use cases of your application

This article aims at presenting several ways of dealing with a request's loading or error in Angular, with regards to these three aspects.

Why should we care about showing error messages and loaders

The first main reason for having a hard time on this, is because we care about our users. We want them to understand easily what is going on the application, and we don't want them to click ten times on a button while waiting for an async operation to be performed.

Thus, it is important to give them as many feedback as we can when we perform async stuff.

RxJS

Angular asynchronous execution management essentially lean on reactive programming, using RxJS. Thus, every pattern presented here involve this library.

In this article, I will mention Observables and BehaviorSubjects , if you are not familiar with reactive programming and especially with these 2 objects, I invite you to check out the very well designed documentation of the library!

Managing one request state using the component state

Principle

The first and easiest way to keep tracks of your requests is to directly watch it within your component.

Let's say we build an api for displaying cute little pictures of cat breeds.

Our component declares two boolean attributes for storing the current state of the request (loading and error) and these attributes are updated throughout the data fetching operation:

What it looks like in a component

@Component({
  selector: 'app-fetch-breeds',
  templateUrl: './fetch-breeds.component.html',
})
export class FetchBreedsComponent {
  constructor(private myService: MyService) {}

  loading = false;
  error = false;
  errorMessage = '';

  fetchBreeds() {
    // Reset request error state data
    this.error = false;
    this.errorMessage = '';

    // Set the component in a loading state
    this.loading = true;

    // Start the fetch event pipeline involving :
    this.myService.fetchBreeds().pipe(
      // A handler which is called when our service triggers an error and
      // which is dedicated to setting the error in a corresponding state
      catchError(err => {
        this.error = true;
        this.errorMessage = err.message; // Or whatever error message you like
      })
      // A callback which is always called whether an error has been triggered
      // or not.
      // It is responsible for setting the component in a non-loading state.
      finalize(() => {
        this.loading = false;
      })
    )
    .subscribe();
  }
}

<div>
  <!-- If we are fetching the data, we display a loader -->
  <app-loader *ngIf="loading"></app-loader>
  <button (click)="fetchBreeds()">See these magnificent cats!</button>
  <!-- If there is an error (and we are done waiting for a response), we display it -->
  <app-error-message *ngIf="!loading && error">
    {{ errorMessage }}
  </app-error-message>
</div>

Managing request state within a service

Why delegating this logic to a service ?

Using services in an Angular application offers many advantages (DRY principle, efficient testing among others). In our case, it also allows us to persist and share data between components / other services.

It also allows abstraction: your component is dedicated to the UI display while a service handles any inner complex logic of the component.

Case 1 - The state is shared between multiple components

Let's say we have two components offering two features based on the cats entity: on one page we see the pictures of the breeds and on the other page we can see some technical details about these breeds.

This architecture involves the duplication of the logic responsible for sending the request, catching errors, and handle a loading state in between.

This logic could be moved into a dedicated service, which will expose one observable for the loading information and one for the errors:

@Injectable()
export class CatBreedsService {
    constructor(private http: HttpClient) {}

    private _loadingSubject = new BehaviorSubject<boolean>(false)
    isLoading$: this._loadingSubject.asObservable();

    // We expose observables because we want our components to have a read access
    // but no write access to the information

    private _errorSubject = new BehaviorSubject<string | null>(null);
    erros$: this._errorSubject.asObservable();

    fetchBreeds(): Observable<CatBreed[]> {
        this._loadingSubject.next(true);
        this._errorSubject.next(null);
        return this.http.get('/cat-breed').pipe(
            catchError(err => {
                this._errorSubject.next(err);
                return of([]);
            }),
            finalize(() => this._loadingSubject.next(false))
        )
    }
}

Now, the components only have to call the fetch helper from the service, and the loading / error state information about the entity will be shared across the entire application.

@Component({
  selector: "app-list-cat-breeds",
  templateUrl: "./list-cat-breeds.component.html",
})
export class ListCatBreedsComponent implements OnInit {
  constructor(private catBreedsService: CatBreedsService) {}

  isLoading$ = this.catBreedsService.loading$;
  hasErrors$ = this.catBreedsService.errors$;

  ngOnInit() {
    this.catBreedsService.fetchBreeds.subscribe();
  }
}

Managing multiple request states within a service

Case 2 - Concurrency Issues

Example use case: multiple occurrences of an autocomplete input

You might also encounter the case where you have to perform a request for the same operation but in multiple contexts :

For instance, let's imagine we have a form where each input dynamically fetch autosuggestion data on change, and show a loader while the update is performed. If we create unique subjects for handling the state of this request, there will be concurrency issues.

A concrete exemple of this in our context would be a page where we can attribute breeds to cat pictures.

The problem with side effects

In this case, since there is only one loading subject for every input, a change on any input will put all the other ones in their loading state or error state.

The same happens with the suggestions observable. It triggers unnecessary renderings in your application, which may have catastrophic consequences for the performance of the application.

Side effects mean you have to break down this component into multiple parts

One could handle this issue by having a complex state including a loading observable and an error observable. The code beneath shows such an implementation :

export interface InputState {
  loading$: BehaviorSubject<boolean>;
  error$: BehaviorSubject<string >;
}

@Component({
  selector: 'app-cat-breed-form',
  templateUrl: './cat-breed-form.component.html',
})
export class CatBreedFormComponent implements OnInit {
  inputs = ['input1', 'input2', 'input3'];
  inputsState: { [key: string]: InputState } = {};

  ngOnInit() {
    this.inputs.forEach(input => {
      this.inputsState[input] = {
        loading$: new BehaviorSubject(false),
        error$: new BehaviorSubject<string | null>(null)
      };
    }, {});
  }

  setLoading(inputId: string, value: boolean) {
    this.inputsState[inputId].loading$.next(value)
  }

  setError(inputId: string, value: string | null) {
    this.inputsState[inputId].error$.next(value)
  }

  getLoading(inputId: string) {
    return this.inputsState[inputId].loading$.asObservable()
  }

  getError(inputId: string) {
    return this.inputsState[inputId].error$.asObservable()
  }

  updateField(inputId: string) {
    this.setLoading(inputId, true)
    this.project$.pipe(
      take(1),
      catchError(err => {
        this.setError(inputId, err)
      }),
      finalize(() => this.setLoading(inputId, err))
    ).subscribe(() => { // do your stuff })}
  }
}

A few remarks :

The component logic is getting intricate.
You can't deport the fetch logic into a service, or at least you have to manage the loading/error state from within the component.
In order to have a readable template, one will probably have to implement getters/setters

thus increasing the code written in the component ts file.

Breaking this down into simpler components, which in this case means implementing a field component, we are back to the first case with no concurrency.

We can create a dedicated service which exposes a registration function which :

takes an id in input
creates a formControl on which the 'change' event triggers a search
returns the formControl and the associated loading, error, and autocomplete suggestions observables.

@Injectable()
export class SearchService implements OnDestroy {
  constructor(private http: HttpClient) {}

  subscriptions: Subscription[] = [];

  registerControl<T>(id: string, initialvalue: T) {
    const control = new FormControl(initialvalue);
    const _loadingSubject = new BehaviorSubject<boolean>(false);
    const _errorsSubject = new BehaviorSubject<string | null>(null);
    const _suggestedSubject = new BehaviorSubject<T[]>([]);
    this.subscriptions.push(
      control.valueChanges
        .pipe(
          switchMap((query: string | null) => {
            if (query !== null && query.length > 0) {
              return this.searchOnQuery<T[]>(query).pipe(
                catchError(err => {
                  _errorsSubject.next(err);
                  return of([]);
                }),
                finalize(() => _loadingSubject.next(false))
              );
            }
            return of([]);
          }),
          tap(suggestions => _suggestedSubject.next(suggestions))
        )
        .subscribe()
    );

    return [
	control,
	_loadingSubject.asObservable(),
	_errorsSubject.asObservable(),
	_suggestedSubject.asObservable()
    ];
  }

  private searchOnQuery<T>(query: string) {
    return this.http.get<T>('/search', {
      params: {
        query
      }
    });
  }

  ngOnDestroy() {
    this.subscriptions.forEach(sub => sub.unsubscribe());
  }
}

Now we just have to register any new input within this service :

@Component({
  selector: 'app-cat-breed-autocomplete-input',
  templateUrl: './cat-breed-autocomplete.component.html',
  styleUrls: ['./cat-breed-autocomplete.component.scss']
})
export class CatBreedAutocompleteInput implements OnInit {
  @Input() initialValue: Entity;
  @Input() autocompleteId: string;

  control: FormControl = new FormControl();
  loading$: Observable<boolean> = of(false);
  error$: Observable<boolean> = of(null);
  suggestions$: Observable<boolean> = of([]);

  constructor(private searchService: SearchService) {}

  ngOnInit() {
    const [control, loading$, error$, suggestions$] = this.searchService.register<CatBreed>(
      this.autocompleteId,
      this.initialValue
    )
    this.control = control;
    this.loading$ = loading$;
    this.error$: error$;
    this.suggestions$ = suggestions$;
  }
}

These 4 observables are now available in the template and can be used with async pipes.

This allows every input to behave independently, since they have their own event management system through the FormControl API

Conclusion

In this article we have seen two different ways of handling the loading state of a request using RxJS, and how to incorporate this within your Angular architecture.

With only one loader to handle in one component, it is okay to implement the UI logic in the component. With multiples loaders at the same time, things become easily messier and we saw how to handle this new complexity in the case of multiple autocomplete inputs.

How to create a video calling app using webRTC and websockets

Fri, 22 Jan 2021 00:00:00 GMT

With lockdowns and restrictions, the past year has not been great for social interactions. This might be the perfect time to build your first video call app, or to add a video chatting feature to your existing project.

If you are unsure on how that works or are having some troubles setting it up, you came to the right place. In this article I will show you how to create a very simple video calling application using react (yes, with Typescript) and SpringBoot. If you are not using this tech stack, fear not, the principles are the same and will apply to whichever framework you want to use.

At the basis of a video call app there are two things we need to use and learn: peer-to-peer network (P2P) and websockets. Let's dive right into them!

What is a peer-to-peer network (P2P)?

You are probably familiar with client-server networks. In this model, computers are connected to a server and can only communicate to it (send or receive data). If two computers (A and B) wants to share data, they have to pass through the server:

This seems a bit inefficient, wouldn't it be easier to cut off the middleman and just let A and B talk to each other? Yes, it would, and this is what is called a peer-to-peer communication. The information is directly shared by the computers:

This can be achieved very easily using WebRTC (Web Real-Time Communications). WebRTC is an open-source framework that allows real time communication between browsers. It was released in 2011 and is supported by Google Chrome, Safari, Mozilla Firefox, Microsoft Edge, Opera, Brave and Vivaldi.

There are also javascript libraries that simplify the creation and usage of webRTC connections. The most popular ones are easyRTC, PeerJs, RTCPeerConnection, SimpleWebRTC and SimplePeer. As shown below, simple-peer is by far the most used one, and it is the library that we will use for the rest of this tutorial.

A webRTC connection takes place with a process called signaling:

Peer A (the initiator) decides to create a connection, an offer signal is automatically created
The offer is given to peer B
Peer B connects to it to create an answer signal
The answer is given to peer A
Peer A connects to it to complete the peer-to-peer connection

What exactly are these offers and answers? These are simply objects which contain two parameters: type and sdp. The type defines whether the object is an "offer" or a "answer". The sdp (Session Description Protocol) contains the information about the connection, such as the IP Address, the port number, and the media to be shared.

If you haven't already asked yourself, there is a slight problem with the process described above. How will user B receive the offer? And how will user A receive the answer? This is where websockets come into play.

What is a WebSocket?

WebSocket is a communication protocol, just like the famous http. The main difference is that while an http protocol is unidirectional, a webSocket is bidirectional. Let's see what this means and how they differ.

With http, a user can make a request to a web server (http://example.com). When this is done a connection is created, the server gives back a response, and once the response is received the connection is closed. This protocol is unidirectional because the sender has to create a request to receive a response. If new information is available, a new request has to be made to obtain it.

With webSockets (ws), a user can also make a request to a web server (ws://example.com). The difference is that when this is done a handshake is made between the user and the server which creates a connection that will never be closed, unlike the http. This ever-lasting connection can be used for bidirectional communication (without requiring the user to always have to make a new request to open a new connection). If new information is available after the original request, it will be sent to the user through the connection.

This is a very important difference, crucial when we want to have access to real-time data. When checking the score of sport match online, nobody wants to have to refresh the page every 5 second (to make an http request and open a new connection) to check whether the score has changed. With a webSocket every time the score changes the server will automatically update it on your browser.

This is exactly how we will share the webRTC offer and answer signals. Whenever user A (the initiator) creates an offer, he will send it through the websocket connection and it will automatically be received by user B (who will then do the same with the answer payload).

The final architecture

The diagram below shows what the final architecture of the app will be.

The two peer signal each other by sending and receiving the offer and answer signals. This will be done using the WebSocket. After this signaling process, the peer-to-peer connection is established and the two peers can share the webcams media streams to video call each other. Now let's get started.

Setting up the WebSocket

The first step is to create the webSocket server, in our spring boot application project. SpringBoot has a sping-WebSocket module which makes the whole set-up extremely simple as we will see.

In the pom.xml file, add the spring-boot-starter-websocket dependency as follows:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-websocket</artifactId>
</dependency>

Now we can create a class annotated with the @Configuration and @EnableWebSocket:

@Configuration
@EnableWebSocket
public class WebSocketConfiguration implements WebSocketConfigurer {

    @Override
    public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
    registry.addHandler(new SocketHandler(), "/videochat").setAllowedOrigins("\*");
    }
}

The class implements WebSocketConfigurer and overrides the registerWebSocketHandlers method, which is used to add websockets endpoints. In this case we add a websocket endpoint "videochat" and with a handler given by the SocketHandler class. The latter, which we will create next, defines how all the sessions and messages coming into ws://localhost8080/videochat should be handled.

The final step is to create the class SocketHandler with annotation @Component:

@Component
public class SocketHandler extends TextWebSocketHandler {

  List<WebSocketSession> sessions = new CopyOnWriteArrayList<>();

  @Override
  public void afterConnectionEstablished(WebSocketSession session) {
    sessions.add(session);
  }

  @Override
  public void afterConnectionClosed(WebSocketSession session, CloseStatus status) {
    sessions.remove(session);
  }

  @Override
  public void handleTextMessage(WebSocketSession session, TextMessage message)
  throws IOException {
    for (WebSocketSession webSocketSession : sessions) {
      if (!session.equals(webSocketSession)) {
        webSocketSession.sendMessage(message);
      }
    }
  }
}

Let's see what is going on above. The class extends TextWebSocketHandler for which we are overwriting three methods:

afterConnectionEstablished is called every time a browser completes a handshake with the webSocket and creates a connection/session (remember this connection will last forever until either parties closes it). The session is appended to the sessions list.
afterConnectionClosed works similarly: when a browser closes a connection this method is called and the session is removed from the sessions list.
handleTextMessage is called whenever a particular session sends a message to the webSocket. When this happens we will iterate through all the sessions connected to the webSocket and send the message to each of them except for the one whose message came from (to avoid sending a message to yourself).

Making the peer connection

Now that the websocket server is done, we can move on to the frontend to build the actual video calling application. We will create a component which return the following content:

enum ConnectionStatus {
 OFFERING,
 RECEIVING,
 CONNECTED,
}

export const VideoCall = () => {
  const videoSelf = useRef<HTMLVideoElement | null>(null);
  const videoCaller = useRef<HTMLVideoElement | null>(null);
  const [connectionStatus, setConnectionStatus] = useState<ConnectionStatus | null>(null);
  const [offerSignal, setOfferSignal] = useState<SignalData>();

  return (
    <div className="web-rtc-page">
      {connectionStatus === null && (
        <button onClick={() => sendOrAcceptInvitation(true)}>CALL</button>
      )}
      {connectionStatus === ConnectionStatus.OFFERING && (
        <div className="loader"></div>
      )}
      {connectionStatus === ConnectionStatus.RECEIVING && (
        <button onClick={() => sendOrAcceptInvitation(false, offerSignal)}>
          ANSWER CALL
        </button>
     )}
     <div className="video-container">
       <video ref={videoSelf} className="video-block" />
       <video ref={videoCaller} className="video-block" />
     </div>
   </div>
 );
};

The page contains two video elements for the user and for the receiver, these have a useRef hook each, which will be used later on to dynamically add the webcam streams. The page also contains a state for the connection status, based on which some elements will be conditionally rendered:

null: when the page is first rendered. A button is present which can be clicked to start the call. This calls the function sendOrAcceptInvitation on click which we will define later
offering: when a user has started a call and is waiting for the answer on the other side. A loader is displayed while waiting.
receiving: when the user receives a call. A button is present to answer the call, this also calls sendOrAcceptInvitation on click, but with different arguments
connected: when the peer-to-peer connection is completed

Now it's time to create the function sendOrAcceptInvitation:

const [simplePeer, setSimplePeer] = useState<Instance>();

const sendOrAcceptInvitation = (isInitiator: boolean, offer?: SignalData) => {
  navigator.mediaDevices
    .getUserMedia({ video: true, audio: false })
    .then((mediaStream) => {
      const video = videoSelf.current;
      video!.srcObject = mediaStream;
      video!.play();

      const sp = new SimplePeer({
        trickle: false,
        initiator: isInitiator,
        stream: mediaStream,
      });

      if (isInitiator) setConnectionStatus(ConnectionStatus.OFFERING);
      else offer && sp.signal(offer);

      sp.on("signal", (data) => webSocketConnection.send(JSON.stringify(data)));
      sp.on("connect", () => setConnectionStatus(ConnectionStatus.CONNECTED));
      sp.on("stream", (stream) => {
        const video = videoCaller.current;
        video!.srcObject = stream;
        video!.play();
      });
      setSimplePeer(sp);
    });
};

In this function we create the simple-peer instances. The arguments of the function are isInitiator (true for the user making the call and false for the user receiving the call) and offer (the offer with the sdp of the caller, only present for the user answering).

The first step is to access the webcam media stream and set it to the videoSelf reference. Next a simplePeer instance is created, with the webcam video as the stream and with initiator value set to true/false depending if this is the caller or the recipient.

The simple-peer instance has certain events which can be fired, for which we will define the respective methods. In this case we have:

signal: when the simple-peer has a signal ready to be sent (the offer or answer). These are the signals that we have to give to the other users though the webSocket (step 2 and 4 of the signaling process). Remember that for the initiator this signal (the offer) is created automatically. A non-initiator, however, needs to connect to the offer signal in order to generate the answer (step 3). This is exactly what the line sp.signal(offer) is for
connect: when the connection between two peers is complete
stream: when a stream is received from the other peer. In this case, each peer will receive the webcam stream of the other user which was passed during the initialisation of the simplePeer instance. Once this is received, we set it to the videoCaller reference

We are almost there, the last step is to create the connection to the webSocket and define what should be done when the signals are received.

const webSocketConnection = new WebSocket("ws://localhost:8080/videochat");

useEffect(() => {
  webSocketConnection.onmessage = (message: any) => {
    const payload = JSON.parse(message.data);
    if (payload?.type === "offer") {
      setOfferSignal(payload);
      setConnectionStatus(ConnectionStatus.RECEIVING);
    } else if (payload?.type === "answer") simplePeer?.signal(payload);
  };
}, [simplePeer]);

Creating the connection to the websocket is very straightforward, we simply create an instance of the WebSocket class, with the endpoint url as its argument.

We can use the onmessage property to catch all of the incoming offer/answer signals (which were sent using webSocketConnection.send() in the sendOrAcceptInvitation function). When receiving the offer we set the signal as a state and change the peerConnectionState, this will render the acceptance button and the user will be able to connect to the signal and only if desired. When receiving the answer signal, however, we automatically connect the initiating simple-peer to the other: successfully closing the connection.

Congratulations! The peer-to-peer connection is now established and the users are able to talk to each other despite the lockdown.

Wrapping it all up

As you saw it is relatively easy to create a video calling application. We were able to create a simple one in just about 100 lines of code! With simplePeer.send and simplePeer.on("data", ()=>{}) we can even send/receive messages and upgrade our application to a video calling AND messaging app.

I hope the tutorial was useful, you can find the full code here.

Why Rome Tools Isn't Ready to Replace Eslint, Webpack and Babel... Yet

Fri, 18 Dec 2020 00:00:00 GMT

Introduction

While building your React, Angular, or Vue project, you may have wondered why you needed to set up and configure Babel, Webpack, ESLint, and Prettier separately and tried to look for an existing bundled implementation of all the features these tools provide. That is when you might have stumbled upon Rome Tools. I will show you in this article some of the best features Rome provides compared to ESLint as well as how I tried migrating my ESLint project to Rome:

TLDR

Rome has great features including explicit diagnostics, provides a VSCode extension which makes it a promising tool. However, only the linter and formatter are available and stability is a major pain point. In addition to this limitation, the lack of configuration options can scare off developers.

The Rome philosophy

Rome's ambition is to unify tools that have previously worked as separate. It wants to have strong conventions with minimal configuration. It aspires to go further than create-react-app or vue-cli as these tools just include multiple other tools with their dedicated configuration files. Rome aims at providing a single frontend development toolchain for all the following features:

Bundling
Compiling
Documentation Generation
Formatting
Linting
Minification
Testing
Type Checking

So far, only the linter is supported by Rome Tools and it was built to provide features that ESLint has been lacking for some time, that is displaying diagnostics. I've used ESLint in many projects and I've always been annoyed by how poor the error messages displayed are. Rome tries to tackle that and I find that the diagnostics displayed with rome check are really helpful to understand the error, how you can fix it, and what you can do to avoid it appearing in the future:

Rome does a great job at displaying diagnostics!

Rome also includes a watch mode, and review to allow you to apply safe fixes, comment to disable the rule, ignore the issue, etc... all of that in its comprehensive CLI:

Diagnostics, advice, and fix options

Rome is also shipped with its file history. Say you apply an autofix on all your files but forgot to exclude a directory, you can easily recover your formatted files with rome recover pop. I find this command truly handy, especially when setting up your formatting or auto-fixing your issues.

Experimenting and migrating from ESLint

To build up a better judgment of Rome Tools, I took one of my existing projects that were using ESLint and tried migrating it to Rome which made me encounter most of the problems Rome has.

The first thing I did was to add Rome to my project. Applying formatting is a cakewalk with Rome: running rome check --apply and Rome will format and apply safe fixes to your entire codebase. I then started working on fixing the linting errors and that's when Rome hit me hard. Most of my React components were written as:

const Component = (props: Props) => {};

Rome has a rule against this (useFunctionDeclarations). It requires that you use function declarations instead of constant declarations:

function Component(props: Props) {}

I'm not going to go into details but I find that first syntax handy. In most of my projects, it allows me, at a glance, to quickly distinguish utils functions from components that return JSX. So I decided I wanted to remove that rule and that's when I discovered that configuring your linting rules is impossible to do with Rome. The contributors chose to go with a zero-configuration stance. The only thing that can be configured is the paths you want to ignore in the configuration file:

lint: {
  ignore: ["src/__mocks__/"];
}

You can also suppress a rule on a specific line with an explanation but it's far from ideal as these comments will undermine actual explanations comments in your code. The lack of configuration was my first pain point while experimenting with Rome.

I still tried to fix all my linting errors while working with Rome with the CLI. While it seems promising, there are yet too many errors that cannot be fixed. Hence, the review CLI tool can prove a little useless when the only option it's offering is to comment out the error.

I also had the case where the tool would present me with the same error over and over again until I realized the suppression comment was just not working properly:

return <Style.Animate className={animation}>
		// rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading
		<BaseComponent {...props} />

Rome offers an extension to use in one IDE: Visual Studio Code (there is no plan to bring the equivalent to Webstorm for the moment). I was really happy when I discovered that because I rely a lot on my IDE to apply formatting, quick fixes, read lint errors output, etc...

Overall, I can say that the extension does also an excellent job at displaying diagnostics. The information that is available in the CLI is also available when hovering on an error:

Explicit diagnostics straight in VSCode

Most of the safe fixes, as well as auto-formatting, can also be applied straight from your IDE without having to use the CLI, which I greatly appreciated. There are a few exceptions to this rule, for instance, I was not able to format my tsconfig.json using the extension but could do it from the CLI.

However, I've been struggling a lot with that extension as well as with the CLI while migrating from ESLint to Rome as they both appear to lack stability. I had more than 200 files to migrate to Rome and I ran into a huge amount of crashes while doing so which made my experience with Rome pretty painful. I find it discouraging to have to restart my IDE or the CLI review because I keep getting exceptions:

! Rome exited as this error could not be handled
and resulted in a fatal error. Please report if necessary.

✖ Connection died

Looking at the number of issues related to the extension or the CLI, we can tell that stability is a major area of improvement for Rome.

Conclusion

I find it amazing that projects such as Rome emerge to tackle the weaknesses of predominant tools such as ESLint and offer alternatives. Rome tries to do that and I can only encourage these kinds of initiatives.

The fact that only the linter/formatter is available isn't a major issue in my opinion. Even if it means that Rome isn't a bundled implementation yet, it is a choice they made to have a part of Rome ready as quickly as possible. Nevertheless, it lacks both configurability and stability. I would consider that point a red flag due to how it slowed me down while migrating. Rome wants to replace all the other tools but, paradoxically, makes it harder by not providing configurability.

Finding out why your code isn't working and how you can improve your development practices is something Rome definitely provides but even with that, I cannot recommend starting a project with Rome at this day. I'm hoping that the Rome developers will stabilize it, offer configurability and implement the rest of the features Rome wants to provide to offer a developer-friendly monolith.

Get started with Django and Jupyter Notebooks on VSCode in minutes

Thu, 17 Dec 2020 00:00:00 GMT

This is a guide primarily for developers familiar with Django, looking to add a data analytics or data science element to their web development project.

Don't have much time and comfortable with Jupyter? Click here to see the initializer file you should create + import into every python script you want to run: Final Code

Want to see an example data analytics script you can use for any Django project? See here: Example Script

Data Science for Web Apps

Django is a (very popular) Python-based framework that helps you build web apps frighteningly quickly. A quick Google/YouTube search for Django tutorials and you'll have an avalanche of content coming your way.

Jupyter Notebook is a tool that lets you build python scripts fast. It's especially popular for data science prototyping as it lets you break up your code and re-run parts of your script.

Why merge the two?

Today, data analytics is (and should be) the mantra behind every successful product owner's launch plan - and as a developers, providing guidance and infrastructure to enable this is extremely valuable. However, combining these two in one stack can lead to some interesting errors and so, I've written this article to help you cross these hurdles quicker than I did.

Prerequisites

At this point, you should have a Django development server setup in a virtual environment (if not, there's plenty of tutorials for that). This setup uses MacOS/OX (though OS shouldn't matter) with VSCode.

If you don't have Python already set up in VSCode with a good formatter and would like to, click here.

Configuring Django to run Python scripts without the server

Start by creating a folder inside your Django root directory named python_functions. All your python scripts can live in here.

At this point, if you try and import something from Django (and you don't have your development server running), you'll get an error that your settings are not configured. To see the error, type the following into hello.py and run your script:

  from django.contrib.auth.models import User

You should see an error message that ends with:

  django.core.exceptions.ImproperlyConfigured: Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

This happens because without running manage.py, Django doesn't configure and run your project - and so doesn't know where to import modules from.

To fix this, we'll need to firstly specify an environment variable to our settings file and then call the setup function. As we'll have to do this for every python script we write, let's do all the setup in a separate script, and import it into hello.py.

Creating an initialiser script

Create a new file in the python_functions directory called django_initializer.py. In this file, we first set an environment variable pointing to our settings.py file, then run the function to setup Django.

But we're not done yet! As django_initializer.py isn't in the project's root directory, python won't recognise "arcwebsite.settings" as a legitimate path. To fix this, we need to manually add the project root directory to the python path.

Altogether, here's what we've got in django_initializer.py:

  # A script that's needed to setup django if it's not already running on a server.
  # Without this, you won't be able to import django modules
  import sys, os, django

  # Find the project base directory
  BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

  # Add the project base directory to the sys.path
  # This means the script will look in the base directory for any module imports
  # Therefore you'll be able to import analysis.models etc
  sys.path.insert(0, BASE_DIR)

  # The DJANGO_SETTINGS_MODULE has to be set to allow us to access django imports
  os.environ.setdefault("DJANGO_SETTINGS_MODULE", "arcwebsite.settings")

  # This is for setting up django
  django.setup()

Remember, this is separate to the actual file we want to run, as seen below:

Make sure the initialiser script is in the same folder as your actual script

Run hello.py once more, and you should see it work with no problems!

Now, you've got a setup to run Python scripts to analyse your data, that can interact with your Django database even when your server isn't running. This is especially useful when prototyping on a local machine before deploying any scripts to production.

The final step is to set up Jupyter Notebook in VSCode, to interact with your Django project.

Setting up Jupyter Notebook in VSCode for a Django Project

Why Jupyter Notebook?

Jupyter Notebook is a tool by Project Jupyter to make data science modelling and prototyping easy. One of the main benefits is the ability to split large procedural programming scripts into individual lines or blocks, written in 'cells', and run them one at a time.

An example use of the cell-by-cell feature of a Jupyter Notebook

Running cells individually has a lot of benefits - for one, you can split your script into data collection, pre-processing, model training, and prediction steps, and run these segments individually. This can be especially useful when tweaking hyper-parameters on your models for example (or for any other instances you may want to re-run just one section of code from your script).

Jupyter Notebooks can of course be run in the browser, rather than in VSCode, using the following command:

  (myenv) $ python manage.py shell_plus --notebook

With this however, you wouldn't get the benefit of other VSCode extensions on your ipynb file, and easy access to your non-python files within the same code editor. Thus, setting up Jupyter inside VSCode makes our dev experience that little bit better.

Installing Jupyter Notebook in Django

So let's get started. First we'll need to install Jupyter and django-extensions. django-extensions is a collection of custom extensions for the Django Framework. It's often needed to support many other custom packages such as this one.

Run this command in your terminal:

  (myenv) $ pip install jupyter django-extensions

Next, add django_extensions to your INSTALLED_APPS in settings.py and restart the django server:

As we're already using VSCode with the Python extension, we can create a new Notebook quite easily. Simply bring up the command bar (cmd+shift+P) and select in 'Create New Blank Jupyter Notebook'.

After opening your first .ipynb file (the file extension for Jupyter Notebooks) you should get a VSCode warning prompting you to install an ipykernel. Click on 'install' when this pops up. Once installed, you won't need to do this again for any subsequent notebook files you create in the same virtual environment.

NB: This will change how your shell looks when you run python manage.py shell on the command line - but unless you've manually customised the default Django shell to your liking, this shouldn't be a problem.

Now, copy over the contents of hello.py (the two imports) into the new .ipynb file, and press run - your code should run without any errors!

Now let's try a simple queryset operation. Add a line to display the first user on your system (or an equivalent queryset operation):

  print(User.objects.all()[0])

There's an error!

Running queryset operations with the default setup causes errors

On closer inspection, you should see the error message ends with:

  SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.

Why does this error occur?

Skip to the next section if you'd just like the fix!

Django was originally built with the WSGI standard in mind - a standard for synchronous Python web servers, frameworks and applications. With the release of Django 3.0 came support for ASGI, a standard for asynchronous applications, allowing developers to create applications designed to run in an async context.

However, certain key parts of Django rely on a global state being maintained, which is not coroutine-aware (i.e. not able to run in an async context). The most prominent example of this is Django's ORM (object-relational mapper) - essentially, its queryset filtering features, which allows you to interact with an SQL database through Python.

Running any of these sync-only Django functions in an async context gives you the SynchronousOnlyOperation error. The ideal way to deal with this is to wrap your function in Django's sync_ to _async(), but for a Jupyter Notebook where the async context is forced onto us by the environment (not our code), we need to manually allow sync-only operations, by setting DJANGO_ALLOW_ASYNC_UNSAFE to true. The Django documentation also has a section explaining async safety and has stated that async support for the ORM will be available in future releases.

Fixing the SynchronousOnlyOperation Error

The quickest way to fix this is by setting another environment variable. Go to django_initializer.py and add:

  os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"

Which means django_initialiser.py in its final state now looks like:

# A script that's needed to setup django if it's not already running on a server.
# Without this, you won't be able to import django modules
import sys, os, django

# Find the project base directory
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

# Add the project base directory to the sys.path
# This means the script will look in the base directory for any module imports
# Therefore you'll be able to import analysis.models etc
sys.path.insert(0, BASE_DIR)

# The DJANGO_SETTINGS_MODULE has to be set to allow us to access django imports
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "arcwebsite.settings")

#  Allow queryset filtering asynchronously when running in a Jupyter notebook
os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"

# This is for setting up django
django.setup()

Finally, reload the window and run the ipynb script again. You should get a success!

Finishing Touches

To recap - we've configured Django to give us access to the framework without the server running and set up a dev environment with Jupyter Notebooks for easy prototyping and testing of data analytics scripts.

With these tools, you should now be able to run Python scripts inside of VSCode with familiar data science packages such as scikit-learn, pandas, and matplotlib, just as you would with a regular Jupyter Notebook - as well as have access to your Django database with queryset filtering.

Good luck!

Start Tracking Website Metrics - An Example

Here's an example script to track the total number of weekly user sign ups on your Django app from a given start date. Note that there are some dependencies required that haven't been mentioned in this tutorial, but after installing them, you should be good to go!

# Imports
import django_initializer

from django.contrib.auth.models import User
from analysis.models import WorkoutEntry
from datetime import datetime, timedelta
import pytz
import time
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)

# Set a start date
start_date = datetime(2020, 7, 13, tzinfo=pytz.UTC)
today = pytz.utc.localize(datetime.today())

# Setup for weekly metrics
week_start = start_date
weeks = []
while week_start <= today:
    weeks.append(week_start)
    week_start += timedelta(days=7)
    week_strings = [week.strftime("%d/%m") for week in weeks]

# Count number of cumulative sign ups
tot_users = [] # total users
pc_growth = [] # percentage growth week on week
for (index, week) in enumerate(weeks):
    weekly_signups = User.objects.filter(is_active=True).filter(date_joined__gte=start_date).filter(date_joined__lt=weeks[index])
    tot_users.append(len(weekly_signups))
    if index > 1:
        pc_growth.append(np.around((100* (tot_users[index] - tot_users[index-1]))/(tot_users[index-1]), 2))
    else:
        pc_growth.append(0)
users_per_week_df = pd.DataFrame({"tot_users":tot_users, "pc_growth":pc_growth}, index=week_strings)

print(np.mean(np.around(pc_growth, 2)))
print(users_per_week_df)

fig, axs = plt.subplots(figsize=(10,6))
axs.plot(users_per_week_df["tot_users"])
axs.set_xticks(axs.get_xticks()[::4])
axs.set_xlabel("Date")
axs.set_ylabel("Total Users")
axs.set_title("Total User Signups",{'fontsize': 18, 'fontweight': 600})
axs.grid()
plt.show()

Setting up Python in VSCode with the Black Formatter [optional]

We'll be adding the Python extension to VSCode, setting a virtual environment and setting up a formatter that you can customise to your preference.

The first step is to ensure a great developer experience for programming in Python in VSCode. For this, you'll first need the VSCode Python extension, which you can find by simply searching for 'Python' in the extension sidebar. As of June 2020, there's also the new 'Pylance' extension, which is also suitable.

Download this and reload the developer window (cmd+shift+P, type in 'reload' and you should see the command) and open a .py file. You should then see the Python version and interpreter in the bottom left corner of the screen. If it doesn't show your chosen virtual environment settings, click on it to update the settings to the interpreter of your choice (select from the dropdown or type in the path required).

Next, we'll set up the formatter - we're going to use Black (Github). As Black is usually installed and run via the command line, we'll have to do some extra work to integrate it into VSCode, and run every time we save a file.

There's no Black VSCode extension, so we'll have to install it the old fashioned way. In your terminal, run:

(myenv) $ pip install black

Next, open your workspace settings (cmd+shift+P, type "settings", and select 'Workspace Settings' without JSON). In the search bar at the top, search 'format on save', then check the checkbox:

Finally, type in 'python formatting provider' and select 'black' from the dropdown:

Now reload your window, create a .py file and each time you save, you should see the words 'formatting with black' flicker on and off on the bottom information bar (in line with your python interpreter details). The next step is optional, and for those who want to customise Black.

When running Black from the command line, you can pass in a list of arguments for custom preferences. To set these, go back into your VSCode settings and search 'black args'. You can then pass in individual strings. For example, to set maximum line length to 100 characters:

That's all you need - now your VSCode should be set up to code in Python, with automatic formatting on save.

Theodo

Spice up Your Website with a DocuSign Electronic Signature Embedded Tool

Introduction

Why this one-page architecture?

Why not using DocuSign emails?

Installation

Let's break it down

Server-side

Client-side

And now the trick!

Conclusion

Useful links

How to develop on Shopify in 2021

Front framework: Liquid

Embed a React App inside Liquid's engine

Shopify's API - Usage and Limitation

Shopify's Admin API

Shopify's Storefront API

Development Workflow: Automatize your deployment with CircleCI, Github and ThemeKit

Conclusion

How to Check Your Website on Mobile in the Local Environment

Chrome device toolbar

It does what you expect and maybe more…

But Chrome device toolbar is not an emulator

Solution evaluation

📱 Close to end-user experience: Mediocre

📋 Access to logs: Excellent

🔧 Easy to set up: Excellent

Using your local network to access your website in the local environment

The principles

How to set it up

How to see the logs

A versatile solution that covers (almost) any use case

But:

Solution evaluation

📱 Close to end-user experience: Good

📋 Access to log: Very good - No Logs

🔧 Easy to set up: Very good

Using LocalTunnel to access your website in the local environment from your mobile with HTTPS

What is LocalTunnel?

How to set it up?

The ultimate solution to local development on mobile?

Pros:

Cons

Solution evaluation

📱 Close to end-user experience: Very Good

📋 Access to log: Very good - No Logs.

🔧 Easy to set up: So-so.

Emulators/simulators

Takeaway

How To Make Tree Shakeable Libraries

What is tree shaking and why is it important?

Realize a library is not tree shakeable in a controlled environment

Use ES6 modules so bundlers can detect unused exports

Use the side effects optimization to make your library side effects free

Preserve the library's module tree and split your code in small modules to fully benefit from the sideEffects optimization

A library module imports a CJS formatted dependency

Code splitting

Do not lose the module tree or the ES modules characteristics when transpiling your library

Babel

Typescript

Use the latest version of a tree shaking capable bundler

Conclusion

Further Reading

An introduction to DynamoDB data modeling

Start from an entity-relationship diagram

Write down the access patterns

Model the structure of the primary key

Visualize your model in NoSQL Workbench

Define secondary indexes if needed

What about sorting?

Wrapping it all up

Icon Library in React: Why Inline SVG Are Better than a Font

So many ways to integrate SVG icons!

SVG as an image

SVG as a background image

SVG as font

Inline SVG

Comparison

Customization

Preserve the library's module tree and split your code in small modules to fully benefit from the `sideEffects` optimization