Theodo

Building Terravision(2021) using Deck.gl and Carto

Tue, 07 Dec 2021 00:00:00 GMT

Adding a detailed and interactive map to a web app can be a time consuming process. Pairing deck.gl and carto with react can make it pain free. Using Carto to manage your data and Deck.gl to render it can make for a great developer and user experience. Resulting in a web app rich with detailed and engaging visualisations.

TLDR: skip to case study section if you're too cool for context...

Intro

At the beginning of the Netflix show The Billion Dollar code, you’re introduced to two German developers, Juri Muller and Carsten Schlute. As the series progresses we learn more about Terravision, an application that the two developers and their team (ART+COM) develop that closely resembles the modern day map software we are all familiar with, Google Earth. The series takes place in the 90s and hence the algorithms and mapping software that we use today has progressed significantly from the first versions of Terravision and Google Earth.

Nowadays, as a web developer, the inclusion of a map in your application can surpass the simple use cases provided by the google maps API. Whether it be needing to display a dynamic view of nearby taxis for your journey home or a heat map showing what people in your city are up to; maps in modern day applications are rich with information and data. Two tools that can be used to build rich map visualisations in web applications are Carto and Deck.gl.

Simply put, Carto is a data warehouse that specialises in data for geospatial analysis with some magic under the hood.

If you’re building an application that requires demographic data detailing the populations of cities in Europe just search through the catalogue, get the URL of the data set and import it using their API. Carto is built with developers in mind and hence they have a great API that pairs really well with the second tool I’ll be talking about, deck gl. If you make a call to their SQL api to obtain the data you’ll see a result containing demographic data mapped to a set of long lat coordinates. This format (GEOJSON) is an industry standard that can be interpreted by most mapping software but can be tedious and cumbersome to generate manually so it’s great to have an easily accessible source.

Note: For readers in the UK another great source of geospatial data is the ONS.

So now we have our maps and the data to enrich them. We need a way to combine them and display the result in our web app. Deck.gl is a software specialising in mapping and geospatial visualisation. It offers a react library designed to create high-performance web-GL based visualisations for large datasets. Hence it’s one of the easiest ways to integrate geospatial visualisations into your applications. The library is really intuitive and highly optimised out of the box. It also contains a bunch of different features that will fit a large number of use cases required by your application.

Case Study

Let’s talk through a real world example. Say you’re an international news company and you want to revamp the user experience of your website. You cover news stories all across the globe but you’ve gotten feedback from users saying finding new, interesting stories can be difficult and your team has proposed a new map interface to display all the current popular stories. As well as displaying the stories on the map you think it would be nice to supplement the articles with contextual information about the countries you’re covering (GDP, HDI, Life Expectancy, etc).

Slight caveat. Unfortunately, Carto is not a free piece of software so for the example I will be using a dataset that I’ve downloaded from Kaggle. Hopefully talking through the full process that I have to follow to integrate a generic dataset into our application and how it differs from using Carto will allow me to highlight the convenience of using Carto in your workflow.

The Data

To begin with you’ll need to source the data for the contextual information. As noted above, a quick search on Kaggle you can find a list of demographic indicators for all countries in the world. Granted this is outdated but this is where something like Carto comes in handy as it provides up to date data in the required format.

Using Deck.gl with next.js and react

Now we have our data, the next task is to set up your project. Deck.gl works with react out of the box so you can use the next project starter to generate a react project and get going from there. Firstly, you’ll need to create a DeckGL component for your page. This is where all of your geospatial visualisations will sit.

<DeckGL initialViewState={INITIAL_VIEW_STATE} controller={true}>
  <StaticMap mapboxApiAccessToken={MAPBOX_KEY} />
</DeckGL>

The above code snippet would display a simple map on the page where you render the DeckGL component. You will see there’s a child component StaticMap. This child handles the map element of our page. Deck.gl can easily integrate with mapbox so all you need to do is create an account and use the public token to add a map to your Deck.gl component.

As you can see the map currently takes the whole of our page but we need to limit it to the hero section of the screen so we can see the articles as well. Adding the following prop to our Deck.gl component allows us to add some styling:

views={[new MapView({ width: '100%', height: '44%' })]}

The resulting page:

So now we have our map displayed on the homepage of our website, the next task is to take the geojson data we downloaded from Kaggle and add it to the map. For this I’ve opted to store the layers in the state which is populated when the page mounts. Using getStaticProps, I read the json file from disk and pass it as a prop to the homepage. This prop is then used to dispatch a redux action that takes the json data as an argument and returns the layers.

Note: The format of the geojson data is not fully inline with what deck.gl expects (one reason why using Carto would be nice here) and hence I have to do some transformation on data load. An alternative solution here would be to process the data once yourself and then have this loaded into the app when the page mounts.

I handle the creation of the layers inside a redux saga. For the geojson data I create a new instance of the PolygonLayer class.

// Create basic polygon layer from geojson data
 const europeLayer = new PolygonLayer({
   id: 'real-poly-layer',
   data: action.payload.geoJson,
   getPolygon: d => (d as { type: string; geometry: { type: string; coordinates: Position[][] } }).geometry.coordinates,
   getFillColor: [160, 160, 180, 200],
   getLineColor: [0, 0, 0],
   getLineWidth: 1,
   lineWidthMinPixels: 1,
   pickable: true,
   stroked: true,
   filled: true,
   wireframe: true,
 })

So we now have an instance of a polygon layer generated from the data read from the disk. The last thing to do now is add it to the Deck.gl component.

layers = { mapLayers };

The result:

As you can see we now have a polygon layer displayed on our map but none of the additional information included in the geojson is visible to the user. Most of deck.gl’s layers can be configured to display tooltips on top of the layer. The pickable prop that I added to the PolygonLayer allows the deck.gl component to access all the additional information and adding a getTooltip prop to your deck allows you to manually configure the structure of your tooltip.

getTooltip={({ object }) => {
if (object) {
    const typedObj = (object as unknown) as {
    properties: { story?: string; scalerank?: number; sovereignt?: string; level?: number; type?: string; geounit?: string; subunit?: string; name?: string; name_long?: string; economy?: string; continent?: string; pop_est?: number; gdp_md_est?: number; income_grp?: string }
    };
    if (typedObj.properties.continent) {
    return `Name: ${typedObj.properties.name_long}\nContinent: ${typedObj.properties.continent}\nGDP: ${typedObj.properties.gdp_md_est}`;
    } else {
    return `City: ${typedObj.properties.name}\nStory: ${typedObj.properties.story}`;
    }
} else {
    return null;
}
}}

Now you can see any specific data that you want when the user hovers over a certain section of the map. Without any need to specify coordinates, boundaries etc. Everything is handled for you.

The last feature we need to add to our map is an icon layer showing where the stories we have on our site come from. In the same saga I mentioned earlier we need to instantiate a new IconLayer class.

// Initial story data that could be pulled in from database
 const pointData = [
   { properties: { name: 'Chavignol', story: '5 Cheese Hacks Only the Pros Know' }, coordinates: [2.844560, 47.397411] },
   { properties: { name: 'Milan', story: '11 Hottest Street Light Trends for 2022' }, coordinates: [9.195585, 45.467905] },
   { properties: { name: 'Kyiv', story: '9 Best Practices for Remote Workers in the Shark Industry' }, coordinates: [30.479160, 50.44044] },
   { properties: { name: 'Zurich', story: '10 Ways Investing in Maple Syrup Can Make You a Millionaire' }, coordinates: [8.519453, 47.385598] },
 ]

 // Icon layer based on the story data above
 const layer = new IconLayer({
   id: 'icon-layer',
   data: pointData,
   pickable: true,
   iconAtlas: 'https://upload.wikimedia.org/wikipedia/commons/e/ed/Map_pin_icon.svg',
   iconMapping: ICON_MAPPING,
   getIcon: d => 'marker',
   sizeScale: 15,
   getPosition: d => (d as { coordinates: Position2D; exits: number }).coordinates,
   getSize: d => 5,
   getColor: d => [50, 168, 82, 255],
 });

The result is a map containing information about both the countries we’ve specified and the stories on our site.

Note: In the real world we would have the icon layer pull from a database so that it’s automatically updated on publishing of a new blog.

Technical Deep Dive

You’ve now seen how easy it is to display rich maps using deck.gl and Carto. I want to take some time to talk about what makes them such interesting pieces of software. In the world of geospatial analysis there are some standard data formats (GeoJSON, Shapefile, KML, CSV, etc) and ArcGis is considered to be at the forefront of the geospatial data handlers. You can think of an ArcGIS server as blob storage with an API for processing and access (although in reality it’s A LOT more complex and feature rich). The ArcGIS server is configured to interpret and process data in a geospatial format and make it easy for developers and analysts to use it where appropriate. Carto can act as both an intermediary or a replacement for ArcGIS. But using it as part of your stack can vastly optimise your load times and improve developer/user experience due to the way it handles your geospatial data.

The Power of the_geom

Let’s talk through another example, you want to create a polygon layer to display on a map of Europe to show the most recent population of each country. To generate this you would need three things:

A map of Europe - https://www.mapbox.com/
The contour data defining the boundaries between countries with a relatively high degree of precision - https://www.kaggle.com/sudhirnl7/human-development-index-hdi
The population data tied to each of the contours for a country - https://www.kaggle.com/sudhirnl7/human-development-index-hdi

So now you have the data, where will you store it? Well for a single country you will likely have millions of data points outlining its boundaries which can amount to ~1GB of data for each country. There are 44 countries in Europe. Now you have to load ~44GB for the user each time they want to see this visualisation. Carto offers a great solution to this. Instead of storing the contour data for a country on a point-by-point basis, it takes the data you’ve scraped from the internet (GEOJSON for example), ingests it and converts it into what they call the the_geom. This compresses millions of data points into one, which can easily be queried, manipulated or converted back into the GeoJSON format all using standard SQL commands. Now you’ve gone from having to load millions of rows of data to only 44.

So we have our data compressed and stored on the Carto servers which can easily be accessed using the Carto API, now we need to figure out how to get it displayed on our map.

More tiles than the kitchen floor

On the homepage deck.gl states it’s a webgl powered framework for large scale datasets. Given how the size of geospatial datasets can quickly grow, having a library that can process and handle large volumes of data on the web is key to building an application with a good user experience. Some of the features I find most interesting are:

Loading Data: Deck.gl is built on top of loaders.gl which is a library the same team built dedicated to handling the loading of large amounts of data. Providing loaders for geospatial, csv, json, vector, image files and more. Leveraging the loaders.gl library means that the load times of large datasets can be kept to a minimum with ease.
Bundle optimisation: Another interesting trait of deck.gl is that it’s built with web developers in mind. Hence features like tree shaking can also be easily configured to keep your bundles as small as possible.
64-bit layers: Given that there are so many use cases for deck.gl the software is built in such a way that it can handle visualisations for many scenarios. The use of emulated 64-bit floating point means that the precision and detail of the visualisations is incredibly high. This means if your data goes down to the centimeter scale then the user will have no issues zooming in to that level.

The End...?

So you've now seen how easy it is to add data rich map visualisations to your web applications. You can see all the code I used to build my demo app here and the link to the actual site TerravisionNews 2021. Any questions feel free to get in touch, PRs/comments welcome.

Comparison of Cloud Run and Lambda to render Blender scenes serverlessly

Tue, 09 Nov 2021 00:00:00 GMT

In a recent post, I explained how to run Blender render jobs on AWS Lambda functions, using Lambda container images. As a follow-up, today I am looking at the same workload running on Cloud Run, the easiest serverless way to run a container on Google Cloud Platform.

Test protocol

vCPUs and RAM

A quick disclaimer is that if you can choose between GCP and AWS, in this case, AWS seems to be a better choice because your Lambda functions can use up to 6 vCPUs whereas Cloud Run is limited to 4. When rendering Blender scenes, increasing CPU power has a huge impact on rendering times. I included a Lambda function with 6 vCPUs in our benchmark and it's about 30% faster than running on 4 vCPU-solutions (be it on GCP or AWS).

The topic of memory is interesting. With Lambda, you cannot independently choose the number of vCPUs and amount of RAM. Based on the amount of RAM you pick, the Lambda will be allocated a certain number of vCPUs. If you pick 10GB of RAM (the largest option available on Lambda) you will get 6 vCPUs. 6.8GB of RAM give you 4 vCPUs, so that's what I went for in order to have the same number of vCPUs on both Cloud Run and Lambda.

With Cloud Run on the other hand, you can independently set vCPU count and RAM amount. The maximum amount of RAM available is 16GB which is significantly more generous than what Lambda offers. Depending on what scene you are willing to render, this can be a crucial point because larger 3D scenes in Blender will require more RAM. So it might happen that you simply cannot render the scene you want on a Lambda with 10GB of RAM. In the use-case that led us to set up this system, I was rendering very small scenes so 10GB were more than enough.

Container image

The container image I am using with Cloud Run is very similar to the one I used on Lambda and described in the previous post. It is also based on the Ubuntu image with Blender that is maintained by the R&D team at the New York Times. The main difference is that I do not install the AWS specific SDK and I run an HTTP server with Gunicorn. The impact of this server on the overall performance of the solution will be minimal since the large majority of the response time is spent rendering the Blender scene.

Region

This recent blog post published by Guillaume Blaquiere shows that not all Google Cloud Platform regions are equal when it comes to Cloud Run performance. I had initially set up our experiment in the us-east4 region and, after reading this, I decided to add a test run in the fastest region. In his article, Guillaume Blaquiere finds out that his code runs 11% faster in the us-west3 region compared to us-east4. I got similar results.

Execution environment

Another variable I took into account is Cloud Run's execution environment. Google gives us the option to either run in a first or second generation execution environment. The second generation is in preview at the moment.

Test scene

For our test, I render a 500-pixel-wide square image of a test scene provided by Mike Pan and available on Blender's Demo Files page. I used Blender's Cycle rendering engine. You can see the output below.

Results

Test settings	Specs	Region	Average duration of 4 function executions (s)
Cloud Run	4 vCPUs, 16GB, first gen	us-east4	79.8
Cloud Run	4 vCPUs, 16GB, second gen	us-east4	81.4
Cloud Run	4 vCPUs, 16GB, first gen	us-west3	77.7
Cloud Run	4 vCPUs, 16GB, second gen	us-west3	71.5
Lambda	4 vCPUs, 6.8GB,	us-east-1	76.1
Lambda	6 vCPUs, 10GB,	us-east-1	51.5

A few observations:

The results with 4 vCPUs are all pretty similar
More vCPUs means noticeably faster render times. That's not breaking news but in our case this really meant Lambda was the way to go.
We can indeed see that Cloud Run does not perform in the same way in all regions. In this case, it performed 14% slower in us-east4 compared to us-west3.
The second generation execution environment of Cloud Run seems to be significantly faster than the first in the us-west3 region but slightly slower in the us-east4 region. Given the fact that this new generation is in preview for now, we can expect things to get better in the future but, for now, test
Using Cloud Run on 4 vCPUs in the us-west3 region is about 6% faster than Lambda with 4 vCPUs in the us-east-1 region.

Next steps

The impact of the region on Cloud Run performance made us wonder if there is a comparable impact on Lambda. So I may update this article with additional information on this.

I would also like to measure the performance impact of using Graviton2 Lambda functions for our task.

It would be interesting to see a serverless solution that lets us use GPUs as this would dramatically speed up rendering jobs. But would the cost of such a solution make it a better solution for our use case? I would have to try!

I will post more article on the topic of serverless Blender in the future. Follow my on Twitter to receive updates!

How to Generate Beautiful PDFs with React and Puppeteer

Wed, 27 Oct 2021 00:00:00 GMT

I recently had to provide a new functionality on my project: the "download as PDF" one. The first thing I wondered was why should we provide this functionality? Doesn't it already exist natively with pretty much all web browsers with a right mouse click / print / "save as pdf" option? Well I tried on my webpage and the result was really disappointing:

I had page breaks in the middle of my data tables, graphs, or any React component which aren't supposed to be split on different pages
I didn't have any title, header, page number except the ugly default ones provided by the browser options
I had many other UX/UI pain points that made the PDF quite unreadable

Example on a Boursorama random page, that looks like:

rendering:

That's when I said "ok this feature may be worth it", but how should I do it? There are many open source libraries that can generate PDFs. But my choice went naturally to the well-known, google-developed library: Puppeteer. According to me, it is the easiest way to deal with PDF generation of Single Page Applications. It may not be so if you don't deal with javascript bundles but with plain HTML/CSS. Indeed, there are easier solutions for this use case like wkhtmltopdf, html-pdf-node or jspdf for example.

In this article, I want to give you a few tips to generate beautiful PDFs of SPAs with Puppeteer. Firstly, I will explain to you how you can create a printable version of your page with React and Puppeteer. Then, I will show you how to use Puppeteer for the generation of your new printable page.

Render a printable version of your page

For this part you don't actually need to have Puppeteer or any other printer service set up. You can make your changes to your code as usual, and then ctrl+P on your page to see what it looks likes:

However, the feedback loop is not as quick as usual.

To adapt your code for printing, you have to bypass the 2 main differences between a web page and a PDF:

The components of a React application are dynamic whereas a PDF is a static file
A PDF has page breaks and a fixed size whereas a webpage is a "one page" app with a variable viewport size

From dynamic webpage to static rendering

Create the "printable version" of your SPA with React. To create the printable version of our page, you will have to add/remove/modify the different components that make up the page.

You basically have 2 solutions for this part:

create a whole new custom design for your page
adapt your current page

If you opt for the second solution (that is less costly), you will have to adapt your existing components. For example, if you have a table with 3 tabs, you will probably want to display the content of all the tabs. Something like displaying the tabs one after the other may do the trick:

Only dynamic:

<Table selectedTabIndex="tab1" />

Dynamic and Static:

const tabNames = ['tab1', 'tab2', 'tab3']

(isPrintable ?
	tabNames.map(tabName => <Table key={tabName} selectedTab={tabName} /> :
	<Table selectedTabIndex='tab1' />
);

In this case, the isPrintable props will determine whether to display the 3 tabs, or just the first one. You may pass this props to every dynamic component of your page, that needs to be adapted for printing.

Deal with page breaks and fixed size with CSS

As you can see with the Boursorama example, your components may be cut off between 2 pages when trying to print your page. It happens because your web browser has no idea where to break page if you don't tell him. This is where the break-inside CSS property steps in. You obviously don't want your previous set of tabs to be cut off in the middle. Neither your graphs or almost any component on your page. Then you would have to adapt the previous code to add this CSS property. It would work with inline-css but you probably don't want to add the style={{ breakInside: 'avoid' }} everywhere in your jsx/tsx files.

You would rather use stylesheets. And instead of adding this property on every CSS class already existing, you'll want to use the media @print option. This will let you custom your webpage for printing only! For example, you may want your text to be a bit bigger or to have a smooth grey color on the printable version, for any esthetic reason or convenience.

We'll just add this in the @media object in your css file:

media @print {
  body: {
    font-size: "16px";
    color: "lightgrey";
  }

  .no-break-inside {
    // apply this class to every component that shouldn't be cut off between to pages of your PDF
    break-inside: "avoid";
  }

  .break-before {
    // apply this class to every component that should always display on next page
    break-before: "always";
  }
}

<MyComponent isPrintable=true className="no-break-inside" />

These few CSS tips should help you improve a lot the rendering of your webpage.

How to use Puppeteer to generate your PDFs

Now, your page is ready for printing. You know it when you pass the isPrintable props to your page, right click + print on your browser, and you are quite comfortable with what you seeing. Here comes the part of printing. You now have a printable version of your webpage, but the users have no idea of it, and even if the ctrl + P on the website, they will see the "dynamic" version of the webpage. To let them generate the PDF version and automate the generation of the latest, you probably want to add a button that will directly generate the PDF server side, and even add some customization. This is what, among other things, Puppeteer is used for.

How Puppeteer works?

Puppeteer is a common and natural way to control Chrome. It provides full access to browser features and, most importantly, can run Chrome in fully headless mode on a remote server [...]

—Dima Bekerman, https://www.imperva.com/blog/headless-chrome-devops-love-it-so-do-hackers-heres-why/


Schema of how Puppeteer works server side

Generation of the React app is done by a web browser. We need the minimal environnement able to execute javascript to render a DOM. Puppeteer will do it by launching a headless chromium. From now on, and since the generation is done on the server, the web browser doesn't need to have a graphical user interface (GUI). Chromium with generate the printable version: the same page the user sees on his web browser but with the isPrintable props activated. Then Puppeteer will execute the pdf function on the page with some custom options that will trigger the printing of the page.

Just add the button with the URL that calls the printer service:

<Button onClick={window.open(downloadUrl, "_blank")}>Download as PDF</Button>

The downloadUrl is actually a GET request on your server that will execute Puppeteer on the server and return content with content-type application/pdf

So what does this Puppeteer code look like?

How to use it?

To be able to actually download the PDF, you just need a few code lines.

The minimal code would then look like:

const puppeteer = require("puppeteer");

(async () => {
  const browser = await puppeteer.launch(); // launch a browser (chromium by default but you can chose another one)
  const page = await browser.newPage(); // open a page in the browser
  await page.goto("https://printable-version-of-my-wbe-page.com", {
    waitUntil: "networkidle2",
  }); // visit the printable version of your page
  await page.pdf({ format: "a4", path: "./my_file.pdf" }); // generate the PDF 🎉
  await browser.close(); // don't forget to close the browser. Otherwise, it may cause performances issues or the server may even crash..
})();

These are the common steps you'll need to generate the PDF. Depending on your backend, you probably don't want not to download the PDF on the server but to render it on a response object, to send it back to the client (the web browser of the user). You should then adapt the page.pdf() method with const buffer = await page.pdf({ format: 'a4'}); and return this buffer on the _blank page the user opened on his browser, waiting for a response.

Add some options to customize the PDF

You can of course adapt the options your naturally have on your browser, like the paper size, the scale, the margins, etc. with the help of the official documentation: https://github.com/puppeteer/puppeteer/blob/v10.4.0/docs/api.md#pagepdfoptions.

One cool option that I recommend, mainly because the default one provided by Google Chrome is really ugly, is the header or footer template. Just read a HTML file template and pass it through the data you want to display such as the current date, the page number for each page, or even an image/logo:

const footerBase = fs.readFileSync("./footer.html", "utf8");

customFooter = footerBase
  .replace("{{date}}", new Date())
  .replace("{{image_data}}", imageData);

await page.pdf({ format: "a4", footerTemplate: customFooter });

using a html template

<style>
#logo {
  height: 40px;
  content: url("data:image/png;base64,{{image_data}}");
}
</style>

<div id="body">
  <div id="page-number-paragraph">
    <span id="date">{{date}}</span>
    <span>Page</span>
    <span class="pageNumber"/></span>
    <span>/</span>
    <span class="totalPages"></span>
  </div>
  <div id="brand-container">
    <span id="logo"></span>
  </div>
</div>

You now have provided to your PDF a fully customized footer.

There are a lot of other options regarding, the PDF generation, but also for the previous steps of launching the browser, opening a new page, going to the URL, that will let you fully customize your PDF generation on the server.

Conclusion

Finally, by adapting your React/CSS code and using Puppeteer, you can easily provide a fully custom PDF of your page. Moreover, Puppeteer is doing all the stuff server side. Which makes this feature fully transparent, quite fast for the end user, and with the same result for every user on any browser! Puppeteer is really powerful and has a lot of options that make the PDF generation quite easy for the developers, and with a rendering much more custom and beautiful than the default one on users' browsers.

A Quick Intro to Elm for React Developers

Mon, 25 Oct 2021 00:00:00 GMT

Elm is probably my favourite programming language - its opinionated but welcoming design provides a compelling case study in how language design can gently guide developers towards writing maintainable code.

Why would a React developer learn Elm?

Elm is a language designed for creating frontend applications, offering an alternative to JavaScript libraries like React. Both were created circa 2012, Elm as an academic thesis project, and React as an industry-backed library. The underlying technology to both is a virtual DOM and a declarative programming style for creating user interfaces. The main difference between the two is that, while React is simply a library built for JavaScript, Elm is an entire domain-specific language, designed from the ground up for purpose.

This means Elm has some language features which are now considered best practice in JavaScript development, such as immutability, lambda functions and type safety. But Elm goes further than React and TypeScript in many ways, with such a powerful type system that it can boast 'No runtime errors in practice', and a significant speed improvement over its competitors.

As TypeScript gains more popularity, I wonder which other Elm features will move into React. My personal hope is that the famously friendly error messages (see image below) will make their way into the language. Elm is a compiled language, so the compiler is re-run after code changes to generate JavaScript for the browser. While this seems like a hindrance, I find the development loop quite engaging: a kind of 'type-driven-development' where the compiler helps guide your work, and gives the developer greater confidence in their code. This 'live-recompile' is made possible due to Elm's lightning-fast typechecking and compiling, something TypeScript is still sorely lacking.

The history of programming language adoption shows us that what's popular isn't always best, as the trending languages of today are often successful by pure coincidence - or as a result of corporate marketing campaigns. I truly believe the industry is moving towards functional programming: React's creator, Jordan Walke, said himself that ReasonML (a functional language closely related to Elm) is 'the language of React'. He posits that, after the success of React, we should once again turn to functional programming and ask 'What else did we miss?'

An Example Project

To help developers familiar with React get to grips with the syntax, I've written the same application in Elm and React, a counter with an increment and decrement button (I've excluded a bit of boilerplate code and import statements from both to simplify things). The page looks like this:

JavaScript:

const App = () => {
  const [count, setCount] = useState(0);

  const increment = () => setCount(count + 1);
  const decrement = () => setCount(count - 1);

  return (
    <div>
      <button onClick={decrement}>-</button>
      {count}
      <button onClick={increment}>+</button>
    </div>
  );
};

Elm:

type Msg = Increment | Decrement

init = { count = 0 }

update msg model =
  case msg of
    Increment -> { model | count = model.count + 1 }
    Decrement -> { model | count = model.count - 1 }

view model =
  div []
    [ button [ onClick Decrement ] [ text "-" ]
    , text (String.fromInt model.count)
    , button [ onClick Increment ] [ text "+" ]
    ]

As you can see, the two programs are similar in many ways, and quite different in others. If you're struggling with the somewhat alien ML-style syntax, check out this quick syntax guide.

To start, we define our Msg type to be either an Increment or Decrement value - more on this later.

After that, the Elm program is split across 3 self-contained function definitions, init, update and view. This contrasts with React, where everything is contained within one function.

The init function is much like the 2nd line of our React app: it defines the initial state of the Model, essentially the datatype for the entire application's state. Unlike when we use React's Hooks, we are encouraged to store all the state for our entire application in one data structure, for reasons discussed later.

The update function then takes two arguments, the Model and a Message. It then decides how to change the model based on the message received, returning a new copy of the model with the count changed. You can read the code { model | count = model.count + 1 } as 'a copy of model where its count is one greater'. Elm calls this function for us whenever a new message is sent, for example on a button press.

Finally, the view function then takes one argument, the Model, and renders it as HTML. Other than the style, this is mostly the same as the return statement in the React function - except that in Elm our buttons send Messages rather than call lambdas. Elm calls view to re-render the page whenever the Model changes.

What benefits does this style provide?

Most of these differences are a result of the fact that Elm is a purely functional language, meaning we aren't allowed to perform any 'side effects' as we would in React with Hooks. Functional purity gives us important guarantees about our codebase - we can be sure that, given the same inputs, we will always get the same output. In a practical sense, this makes the code easier to read - you know the only values that will change in a function are its parameters, so you don't have to waste time reading the whole surrounding context. This is one reason why Elm files can be longer than JavaScript ones without compromising maintainability, as the smallest unit you have to comprehend is a function, not an entire module.

However, separating out the code into init, update and view has other benefits - the separation between code describing the application logic and how it looks is now enforced by the language.

The guide refers to this structure as 'The Elm Architecture' (see diagram above). The Elm runtime manages our application for us, calling view whenever the model changes and update with every message received. The interactions here will be familiar to users of Redux, which was inspired by Elm, where a Redux 'Action' roughly approximates an Elm Message, and update is akin to a 'Reducer'.

Having a centralised Msg and Model type can boost productivity, as changing these types in a large module will cause the compiler to alert us to all the places that subsequently need updating, freeing us from the task of searching through the codebase manually. Centralising the state into a single Model also has architectural benefits. It allows us to make illegal states un-representable, and forces us to thoroughly describe all the dynamic aspects of our webpage. Modelling the state of our application as a data structure in this way is a powerful tool, that helps us consider what really lies at the core of our application.

A stricter type system can seem like a limitation, as an example, where React lets us simply write {count}, Elm makes us write text (String.fromInt model.count), converting the Int to a String to Html. However, as TypeScript evangelists will tell you, the advantages of type safety often outweigh the additional work, as it increases the predictability of the code and, in Elm's case, eliminates the possibility of a runtime error.

Where are these famous types?

In Elm, types can usually be inferred from the code you write, so you almost never need to write type annotations to reap the rewards of type safety. However, they are often considered good practice as they can make the code more readable and help the compiler show better error messages. So, for the sake of interest, here is the above program with type annotations:

type Msg = Increment | Decrement

type alias Model = { count : Int }

init : Model
init = { count = 0 }

update : Msg -> Model -> Model
update msg model =
  case msg of
    Increment -> { model | count = model.count + 1 }
    Decrement -> { model | count = model.count - 1 }

view : Model -> Html Msg
view model =
  div []
    [ button [ onClick Decrement ] [ text "-" ]
    , text (String.fromInt model.count)
    , button [ onClick Increment ] [ text "+" ]
    ]

First, we give the Model a formal type. We use the syntax type alias here, because we're not introducing a new set of values (as with Msg), simply a shorthand for { count : Int }, a record with an integer field called 'count'. We use this new type to annotate the declaration of init, saying it is just a constant of type Model.

Next, we give update a type annotation, Msg -> Model -> Model. This is essentially saying 'update is a function which takes a Message and a Model, and returns another Model'. This is somewhat obfuscated by the fact Elm functions are 'curried' by default, making the actual meaning more like 'update is a function which takes a Message, and returns a function which takes a Model and returns a Model'.

Finally, we annotate view with the type Model -> Html Msg. This states that the view function takes a model and returns some HTML, specifically HTML with our custom Messages embedded within it. This is a form of parametrised type: the TypeScript equivalent would be Html<Msg>.

What next?

If you're interested in experimenting with Elm, you can neatly integrate Elm components into your personal React projects using @elm-react/component.

For those interested in learning more about the language and its influence on frontend development, I'd recommend one of my favourite talks, 'The life of a file' by Evan Czaplicki, Elm's creator.

Thanks to the compiler's helpful and friendly error messages, the language almost teaches itself (after a brief skim of the guide) - so if you're a 'learn-by-doing' type of developer I recommend elm-live to re-run the compiler every time you change your file. It's an interesting, even somewhat calming way of programming, and while the language may not have seen mainstream success yet, I think it can continue to show us a glimpse of the future of web development.

Migrating from React Redux to React Query

Tue, 19 Oct 2021 00:00:00 GMT

React Redux is an extremely popular package, receiving about 4.5 million weekly downloads. Another popular package, Redux Saga, is a Redux side effect manager — it alone receives over 900k weekly downloads. The combination of React Redux with either Redux Saga or Redux Thunk is one of the most common additions to React projects.

However, over the last year or so, React Query has been gaining popularity quickly as an alternative to React Redux. Last year, React Query had only about 120k weekly downloads, but as of October 2021, React Query downloads are nearly six times more frequent, at nearly 700k downloads per week.

React Query has some distinct advantages over React Redux for storing server state.

A major advantage of React Query is that it is far simpler to write than React Redux. In React Redux, an operation to optimistically update a field requires three actions (request, success, and failure), three reducers (request, success, and failure), one middleware, and a selector to access the data. In React Query however, the equivalent setup is significantly simpler as it only requires one mutation (which consists of four functions) and one query.

Another advantage of React Query is that it works more smoothly with navigation tools built into many IDEs. With React Query, you can use “Go To Definition” to more easily reach your mutation whereas with Redux Saga, you must search the entire codebase for uses of the action.

On any existing project, making the switch to use a new package can be a daunting task. Today, to help make this task more manageable, we will be looking at an example where an application using React Redux with Redux Saga will be migrated to React Query.

Our Initial Project

To start, let's take a look at a basic optimistic Redux setup with three reducers, three actions, a selector, and a saga.

In our example, we have a React website where a user can input their name and their name is displayed on the page. A selector is used on the React page to display the user’s inputted name. When a user inputs their name, a request action is dispatched that causes a reducer and a saga to run. The reducer optimistically updates the store with the new value of the name. Meanwhile, the saga makes a call to the backend to save the user’s name.

If the backend returns an HTTP success, the saga dispatches a success action. The success action causes a reducer to run which updates the value of the name in the store.

If the backend returns an HTTP failure, the saga dispatches a failure action. The failure action causes a reducer to run which changes the value in the store back to its value before the optimistic update.

NameComponent.tsx

import React, {
  ChangeEvent,
  FormEvent,
  FunctionComponent,
  useCallback,
  useEffect,
  useState,
} from "react";
import { useDispatch, useSelector } from "react-redux";
import { selectName, updateNameRequest } from "../redux/slice";

export const NameComponent: FunctionComponent = () => {
  const name = useSelector(selectName);
  const [localName, setLocalName] = useState<string>("");
  const dispatch = useDispatch();

  useEffect(() => {
    if (name) setLocalName(name);
  }, [name]);

  const handleChange = useCallback((event: ChangeEvent<HTMLInputElement>) => {
    setLocalName(event.target.value);
  }, []);

  const handleSubmit = useCallback(
    (event: FormEvent) => {
      dispatch(updateNameRequest({ name: localName }));
      event.preventDefault();
    },
    [dispatch, localName]
  );

  return (
    <div>
      <p>{`Your name is ${name ?? "unknown to me"}.`}</p>
      <form onSubmit={handleSubmit}>
        <label>
          {`Enter name: `}
          <input type="text" onChange={handleChange} />
        </label>
        <input type="submit" value="Update Name" />
      </form>
    </div>
  );
};

slice.ts

import { createSlice, PayloadAction } from "@reduxjs/toolkit";
import { RootState } from "..";

export interface NameState {
  name: string | null;
  error: string | null;
}

export const initialState: NameState = {
  name: null,
  error: null,
};

export const nameSlice = createSlice({
  name: "name",
  initialState,
  reducers: {
    updateNameRequest: (state, action: PayloadAction<{ name: string }>) => ({
      ...state,
      ...action.payload,
    }),
    updateNameSuccess: (state, action: PayloadAction<{ name: string }>) => ({
      ...state,
      ...action.payload,
    }),
    updateNameFailure: (
      state,
      action: PayloadAction<{ error: string; name: string }>
    ) => ({
      ...state,
      ...action.payload,
    }),
  },
});

export const selectName = (state: RootState) => state.name;

const { actions, reducer } = nameSlice;

export const { updateNameRequest, updateNameSuccess, updateNameFailure } =
  actions;
export type NameAction =
  | typeof updateNameRequest
  | typeof updateNameSuccess
  | typeof updateNameFailure;
export { reducer as nameReducer };

sagas.ts

import { ActionType } from "typesafe-actions";
import { call, put, select, takeEvery } from "redux-saga/effects";
import {
  selectName,
  updateNameFailure,
  updateNameRequest,
  updateNameSuccess,
} from "./slice";
import { updateName } from "../api/api";

function* updateNameSaga(action: ActionType<typeof updateNameRequest>) {
  const { name } = action.payload;
  const oldName: string = yield select(selectName);

  try {
    yield call(updateName, name);
    yield put(updateNameSuccess(action.payload));
  } catch (error) {
    yield put(
      updateNameFailure({ error: (error as Error).message, name: oldName })
    );
  }
}

export function* nameSagas() {
  yield takeEvery(updateNameRequest, updateNameSaga);
}

store.ts

import { applyMiddleware, createStore } from "@reduxjs/toolkit";
import createSagaMiddleware from "redux-saga";
import { nameSagas } from "../redux/sagas";
import { nameReducer } from "../redux/slice";

const sagaMiddleware = createSagaMiddleware();

export default function configureStore() {
  const store = createStore(nameReducer, applyMiddleware(sagaMiddleware));

  sagaMiddleware.run(nameSagas);
  return { store };
}

Converting the Redux Store to React Query

The first step of our conversion is to create a query to replace the selector and the store. Below is a custom React hook that contains a useQuery to store the user’s name. In this example, getName is a call to the backend to get the name stored in the database.

import { getName } from "../api/api";

export const getNameQuery = () => ({
  queryKey: ["name"],
  queryFn: getName,
});

Converting the Saga to a Mutation

The next step of our conversion is to create an optimistic mutation to replace the saga, reducers, and Redux actions. Below is an optimistic mutation in a custom React hook that optimistically updates the query and invalidates the query after the update.

import { useMutation, useQueryClient } from "react-query";
import { updateName } from "../api/api";

export const useEditName = () => {
  const qc = useQueryClient();
  return useMutation<void, unknown, { newName: string }, string>(
    async ({ newName }) => {
      await updateName(newName);
    },
    {
      onMutate: async ({ newName }) => {
        await qc.cancelQueries("name");

        const currentName: string | undefined = qc.getQueryData("name");
        qc.setQueryData("name", newName);

        return currentName;
      },
      onError: (_error, _variables, context) => {
        if (context) qc.setQueryData("name", context);
      },
      onSettled: async () => {
        qc.invalidateQueries("name");
      },
    }
  );
};

The mutation function in this case calls the backend with updateName to change the name stored in the database.

Before the mutation function, Redux Query executes onMutate. In this case, we cancel the query that we are updating, get the current value for the query, and update the value for the query with the new value. We then return the old value for the query so we can revert if our optimism does not pay off.

If the mutation function throws an error, Redux Query executes onError. All we need to do is set the query to the context parameter if the backend call fails.

Lastly, Redux Query executes onSettled, which simply invalidates the query so Redux Query knows that the data is stale. This forces Redux Query to get the name fresh from the backend.

Final Project

The new query and mutation can be used in the following way in the React component. In the end, we will have a React Query setup with an optimistic mutation. Just two hooks replace the original React Redux and Redux Saga setup.

name.ts

import { useMutation, useQueryClient } from "react-query";
import { getName, updateName } from "../api/api";

export const getNameQuery = () => ({
  queryKey: ["name"],
  queryFn: getName,
});

export const useEditName = () => {
  const qc = useQueryClient();
  return useMutation<void, unknown, { newName: string }, string>(
    async ({ newName }) => {
      await updateName(newName);
    },
    {
      onMutate: async ({ newName }) => {
        await qc.cancelQueries("name");

        const currentName: string | undefined = qc.getQueryData("name");
        qc.setQueryData("name", newName);

        return currentName;
      },
      onError: (_error, _variables, context) => {
        if (context) qc.setQueryData("name", context);
      },
      onSettled: async () => {
        qc.invalidateQueries("name");
      },
    }
  );
};

NameComponent.tsx

import React, {
  ChangeEvent,
  FormEvent,
  FunctionComponent,
  useCallback,
  useEffect,
  useState,
} from "react";
import { useQuery } from "react-query";
import { getNameQuery, useEditName } from "../dataLayer/name";

export const NameComponent: FunctionComponent = () => {
  const editNameMutation = useEditName();
  const { data: name } = useQuery(getNameQuery());
  const [localName, setLocalName] = useState<string>("");

  useEffect(() => {
    if (name) setLocalName(name);
  }, [name]);

  const handleChange = useCallback((event: ChangeEvent<HTMLInputElement>) => {
    setLocalName(event.target.value);
  }, []);

  const handleSubmit = useCallback(
    (event: FormEvent) => {
      editNameMutation.mutate({ newName: localName });
      event.preventDefault();
    },
    [editNameMutation, localName]
  );

  return (
    <div>
      <p>{`Your name is ${name ?? "unknown to me"}.`}</p>
      <form onSubmit={handleSubmit}>
        <label>
          {`Enter name: `}
          <input type="text" onChange={handleChange} />
        </label>
        <input type="submit" value="Update Name" />
      </form>
    </div>
  );
};

The last step for this setup is wrapping the application in a QueryClientProvider.

App.tsx

import React from "react";
import { QueryClient, QueryClientProvider } from "react-query";
import "./App.css";
import { NameComponent } from "./components/NameComponent";

const queryClient = new QueryClient();

function App() {
  return (
    <div className="App">
      <header className="App-header">
        <QueryClientProvider client={queryClient}>
          <NameComponent />
        </QueryClientProvider>
      </header>
    </div>
  );
}

export default App;

React Query is quicker to write, easier to navigate, and, now that you know how to use it, an alternative to storing and updating server state with React Redux with Redux Saga or Redux Thunk.

How to Create a Localized Date Pipe in Angular 📅

Mon, 11 Oct 2021 00:00:00 GMT

TLDR:

To translate dates in Angular, we can think about three approaches: setting the app global locale, passing a specific locale to Angular date through arguments, or creating a custom localizedDate pipe. The localizedDate pipe combines Angular date pipe and the lib ngx-translate. This solution prevents us from explicitly passing the locale as an argument in the template while still being able to dynamically change the dates' locale without reloading the app. Those aspects make the localizedDate pipe impure like [ngx-translate]'s translate pipe.

Use a pipe in the template

We use pipes either in a service or a template (the HTML code). In the following example, we are using the template syntax of the slice pipe and the uppercase pipe.

<div>{{ 'Grogu' | slice:1:4 | uppercase }}</div>
<!-- output: <div>ROG</div> -->

This is equivalent to 'Grogu'.slice(1, 4).toUpperCase() in Javascript.

Sadly, there is only a short list of built-in pipes in Angular. Wait a second, could we create our own?

That's fairly easy to do because a pipe is simply a data transformer.

Different ways to translate dates in a template

Angular provides a built-in date pipe. It transforms a date into a string value (eg. 2021-03-21 -> Mar 21, 2021). The date pipe uses the app locale as its default locale (eg. 2021-03-21 -> 21 mars 2021 for a french locale).

To translate dates in a template, we can think about three approaches:

by setting the default app locale
by passing the locale as arguments of the date pipe
by combining ngx-translate and the date pipe in a new custom pipe

Approach 1: Setting the default locale

The date pipe uses the LOCALE_ID to determine the default locale to use. By default, LOCALE_ID is set to en-US. We can change the app locale by LOCALE_ID setting in the AppModule.

@NgModule({
  providers: [
    { provide: LOCALE_ID, useValue: 'fr-FR' }
  ]
})

With the static string value fr-FR, the LOCALE_ID value can not be dynamically changed. Instead of statically setting the LOCALE_ID value, we can provide a service as a value.

@NgModule({
  providers: [
    { provide: LOCALE_ID, useFactory: appLocaleService },
  ]
})

For changes made to appLocaleService to be reflected in our app, we need to reload the Angular app. LOCALE_ID is required in Angular build process. Reloading the app may reset the app state and re-trigger HTTP requests.

Approach 2: Passing the locale as arguments of the `date` pipe

The locale used in the the built-in date pipe can be set through the pipe's arguments.

<div>{{ '2021-03-21' | date:undefined:undefined:'en' }}</div>
<!-- output: <div>Mar 21, 2021</div> -->
<div>{{ '2021-03-21' | date:undefined:undefined:'fr' }}</div>
<!-- output: <div>21 mars 2021</div> -->

Same as the previous approach, the static locale can be replaced by value resolved in a service. That way, we can dynamically change the locale without reloading the entire app.

The repetitive syntax makes this approach frustrating in the syntax.

Approach 3: Using a custom pipe `localizedDate`

The internationalization library ngx-translate provides a translate pipe that can be used as follows.

<div>{{ 'my.traduction.key.for.hello-world' | translate }}</div>
<!-- output (if the current locale is EN): <div>Hello world</div> -->
<!-- output (if the current locale is FR): <div>Bonjour le monde</div> -->

The goal here is to create a localizedDate similar to translate. It will subscribe to any change in the application locale and behave accordingly.

<div>{{ '2021-03-21' | localizedDate }}</div>
<!-- output (if the current locale is EN): <div>Mar 21, 2021</div> -->
<!-- output (if the current locale is FR): <div>21 mars 2021</div> -->

With the localizedDate, we gain in maintainability in the syntaxe and by using a single internationalization library for dates and contents. As others, it has its cost, this approach adds extra computation in the app. More details next in the article.

Our own custom localized date pipe

The idea is the use the built-in date pipe along with ngx-translate. Sounds complicated? Let's figure that out!

@Pipe({
  name: 'localizedDate',
  pure: false
})
export class LocalizedDatePipe implements PipeTransform {

  constructor(private translateService: TranslateService) { }

  transform(value: Date, format = 'mediumDate'): string {
    const datePipe = new DatePipe(this.translateService.currentLang);
    return datePipe.transform(value, format);
  }
}

Yeah, as simple as that. In this implementation, there is:

the declaration of the new pipe with @Pipe (note pure: false, we will come back to that later)
the injection of the app's TranslateService
the transform method of the PipeTransform interface

In the transform method, we are using the service syntax of DatePipe. Hence, we can specify the current locale during instantiation.

Pure vs Impure pipe

Ahmed Ghoul wrote a super article about Pure vs Impure Pipe in Angular. Pure pipes (like the built-in date pipe) are called only if the pipe's inputs changed. On the other hand, impure pipes are called on every change detection cycle.

The localizedDate can not be pure. The main purpose of this pipe is to avoid passing the current locale as input. Therefore, it has to be an impure pipe to listen to the current locale changes triggered by other components.

For the same reason, the translate pipe from ngx-translate is impure as well.

A Stackblitz is worth a thousand words

The source code of the localizedDate in a concrete example can be found here :

The last word

The approach you may want to choose to translate dates depends on your application and needs.

If you have a specific URL per locale and redirect the user when he changes locale, the first approach should work just fine. If you are already using ngx-translate, creating a localizedDate starts to make sense.

To help you with your decision, here is a quick comparison between the three approaches:

	Syntaxe	Reusability	Reload required	Performance
Setting LOCALE_ID	⭐️⭐️⭐️	⭐️⭐️⭐️	YES	⭐️⭐️⭐️
Using date pipe argument	-	⭐️	NO	⭐️⭐️⭐️
Custom localizedDate	⭐️⭐️⭐️	⭐️⭐️⭐️	NO	⭐️

Happy coding 👨🏽‍💻

How I Made My Life Easier by Letting QA Write E2E Tests?

Mon, 20 Sep 2021 00:00:00 GMT

Lately, I have worked as a web developer with a company that deals with logistics. Their business is to supervise the transport of commodities from point A to point B. The application I worked on was intended for the suppliers of this company to track their merchandise. Each persona would connect to the application and perform tasks such as declaring the dispatch or the reception of the products.

The people who worked on the previous projects followed a V-model lifecycle. These projects were delivered with many bugs after a few years. Developers spent a couple more years trying to fix the application. Therefore, the quality of the application is one of their main focuses. We were using an agile methodology and after each sprint, all the application was tested to ensure that no bug or regression appeared. QA (Quality assurance) was responsible for the quality of the application.

Testing the application - how to do it effectively?

The QA was manually testing the application. Running the full set of tests would take half a day, so the tests were performed every other week. It was not frequent enough to catch all the bugs and meanwhile, QA could not focus on their other tasks. It was not satisfying enough for the client and the QA,that is why the developer team suggested implementing end-to-end (e2e) testing.

But this idea was not perfect: in typical e2e testing, developers are writing the e2e tests. While they are implementing those tests, they are not developing features, which is detrimental for the client. On top of that, the QA, which is responsible for the quality of the application, does not understand what the e2e tests are testing.

Our challenge was finding a solution where QA would not have to test the application manually but without making testing the developers’ responsibility.

What if QA could write tests?

If the QA can write the e2e tests, the tests and the quality of the application are directly managed by the QA. The QA ensures thus the high quality of the application. It is pretty fast for the QA, because they write the test once, and it is then run automatically. And no need to wait two weeks now to run the tests anymore, they can be run every day.

Which means developers can fix bugs faster, for they might have created the bug recently and the code is still in their mind.

This was nice and good, but we had one major issue, for our QA did not have a developer background, yet popular e2e test frameworks, such as Cypress, are asking for knowledge in Javascript.

Writing tests without coding (or very little)

A common test framework, called Cucumber, can be used without development skills and both developers and the QA had already used it. Through its language (Gherkin), it allows team members to write tests in a human-readable language.

Cucumber can be then connected to many common testing tools, and it can be connected to Cypress thanks to cypress-cucumber-preprocessor. To develop tests, all you need is a grammar of test sentences, for example, “When I log in”. You still have to code, but your sentences are reusable, so when the grammar is done right, developers are not needed for every test case.

Our process was as follows. First, the QA would decide which test cases were to be tested automatically (critical test cases that are very tedious to run manually). The developers would then translate from Gherkin to Cypress by creating the Gherkin grammar that the QA can use.

You'll find an example of a feature written in Gherkin by the QA and the steps definitions in Cypress (as the developer will code them). You can retrieve this example here.

# Scenario in Cucumber/Gherkin
Feature: Cypress click
    Scenario: Click and navigate
        Given I'm on the "/" page
        When I click on "Querying"
        Then I should have "querying" in the URL

// Translation of the Gherkin steps to Cypres
import { Given, When, Then } from "cypress-cucumber-preprocessor/steps";

const baseUrl = "https://example.cypress.io";
Given("I'm on the {string} page", (url) => {
  cy.visit(baseUrl + url);
});

When("I click on {string}", (label) => {
  cy.get(".home-list").contains(label).click();
});

Then("I should have {string} in the URL", (url) => {
  cy.url().should("include", url);
});

Make our cucumber tests run

Our team was using Xray, a popular plugin that powers up Jira by providing tools to manage tests. As a result, the QA had to write all its test cases in XRay, which allows the QA to write manual tests or cucumber tests. In addition, Xray comes with an API that developers can use to get test cases or send tests results. Each morning, the tests were automatically pulled from XRay, run and then the results were sent to XRay.

Example of a test case in XRay

How was this solution for us?

With this solution, developers do as little code as possible. They only need to translate the Gherkin sentences into Cypress tests, but each sentence is reusable, so the QA can write new tests without the need of development.

The QA has a complete picture of the quality of the application. They decide which test cases should be executed each day and they know immediately if a test fails.

This process is not perfect and it still requires some maintenance from the developers, but no more than for regular automated tests. It even requires less time from developers as they are not responsible for keeping the test cases up-to-date.

And if you want to have your QA writing the e2e tests?

First, train your QA for using Cucumber. It is easy to understand, even with no development skills. Feel free to check the comprehensive Cucumber documentation.

Next, you need to decide where the QA should write their tests. Requiring QA to use git may be excessive. Xray is a good tool as it is designed to be used with Cucumber, but it can be any tool that your QA uses to write its test sets. Ideally, the tool should support Gherkin syntax and have an API to let developers run tests automatically.

Architecture guidelines for large Angular applications

Wed, 15 Sep 2021 00:00:00 GMT

TLDR;

When coding an Angular app for the first time, this is dangerous to keep old habits and reproduce patterns that worked for other apps like React or Vue. Working in a very large (~100 devs) project, I know that old habits can make code of poor quality.

I propose some guidelines inspired from Angular's recommendations and Clean Code practices that will help your team to write maintanable code and ease newcomer's onboarding!

Put the code right where it belongs : use the power of services and pipes to unclog your components.
Code in a reactive way your components with services
Avoid passing inputs/outputs all around thanks to the Dependency Injection System.
Split Services when they become too big. Do the same with modules

Old habits that get you wrong

Those patterns you need to let go of

Angular's intimate relationship with Typescript makes it easy to put data logic everywhere. Therefore it's easy to agree that dumb contains view-printing, and smart contains fetching. It is a sound decision to improve your code readability.

In terms of performance and seperation of duties, Angular proposes an idiomatic way to code. But many of us have been tempted to try famous patterns like Container Components (or smart), or trying to reproduce stateless Components of React and other frameworks.

That being said, Data often plays a central role. Once again, the temptation to try famous patterns like functional components and a Redux data-store is very strong. Indeed, some don't see how to find a robust way to manage their data.

Nevertheless, trying to mimick patterns you learned on other codebases blindly into an Angular application is such a dangerous idea and I want to show you how to make your Angular application maintanable.

Improving readability in consulting companies

I have been working here at Theodo (also in the UK) with teams that encoutered a huge developers turnover. Developers rarely stayed more than one year on the same code base. I even found myself switching teams every 2 months.

Therefore, it is crucial to choose a pattern that will help new developers to quickly switch codebases. That's why I want to introduce you to the right way of thinking your Angular Application Architecture. This approach makes small use of other famous patterns and mainly explains Angular's recommendations.

Guidelines for clean and efficient Angular coding

Use angular-cli

These are the rules you can establish in your team to make you Angular application cleaner. And as a quick rule, I advise you to always use angular-cli when creating new components and services in your app, so you're not tempted to write less.

You are writing presentation code in .ts or .html ? Make a pipe

If the data needs simple mappers, create pipes that you will use in component's template. Indeed those mappers add usually nothing but noise to your code, hide them away !

Don't write functions for data transformation in your component's .ts, and don't write complex functions in your component's html template. Code smells :

You call functions in your template (See why that's a bad idea)
You write a lot of logic in {{ your template }}

Here is an example of a too-intelligent component. It has functions in its template, with logic...

<ul>
  <li *ngFor="let member of team.member">
    {{ isUserFrench(member) ? '🇫🇷' : '🚩' }} Star: {{
    isUserCapableOneOfTheBest(member) }}
  </li>
</ul>

You can put everything in a pipe, a test it seperately !

@Pipe({
  name: 'memberBasicInfoList'
})
export class MemberBasicInfoListPipe implements PipeTransform {
  transform(team: Team): {flag: '🚩' | '🇫🇷', isStar: boolean}[] {
     //calculate flag and isStar
     return {flag, isStar}
  }

Then you have a very readable template !

<ul>
  <li *ngFor="let memberInfo of team | memberBasicInfoList">
    {{ memberInfo.flag }} Star: {{ member.isStar }}
  </li>
</ul>

Make reactive Services that support your components business logic

Services are your intelligent blocks of code. If you wanted to put code in a container component, put it here ! Here are some some rules of thumb :

Wrap all the data fetching in a service, and inject them in the component(s). This service should expose ready to use observables. It should also give methods to send data from components to the service. Same rule applies if a service becomes too big : split it and inject one service into the other service.

Code in a reactive way with observables as inputs and funtions as outputs

Use observables in your components as if they were already containing the data you want, because that is not the role of component to update the data. Then build public methods in your service to notify it of the last events. Your component should contain very minimum logic.

Think of your component as a puppet of the service. It shows the data the service wants. It tells the service when something happens.

Split your services when they become bigger than 100 lines

Put business logic as soon as possible in services and prepare the data inside of them to be displayed. For example a TeamMember.service that relies of TeamsStore.service and for example another User.service (that can come from somewhere else than TeamsModule)

To split your code, detect parts of code that are heavily linked to each others.

When your base module becomes too big and handles many topics : split into smaller modules

Put common business items in a module, for example "TeamsModule". Split them conviniently with Angular routing as soon as possible. When ?

Some parts become more and more independant, and some parts are never displayed together
Your module is too big for a new-comer to understand it

Your parent/children modules can communicate through common services. Learn more about when to split modules here : Angular: Understanding Modules and Services

Let's study an example: Breaking down one component into smaller simpler pieces

We talked about writing reactive services, and breaking down big chuncks of code into smaller meaningful one. This example shows a typical example when you have a very big component that needs to be refactored. I often found myself in this exact situation.

InviteTeamMemberComponent(httpClient: HttpClient, errorService: ErrorService, router: Router)
A. Can call the team-backend and POST a new team member
B. Will show an error message (in global snackbar) if failed
C. Will redirect to Team dashboard if successful
D. Creates FormGroup for team member informations (also stores the data and validates it)
E. Displays the data in the template
F. Keeps track of the loading state of the query

That's a lot of responsibilities, huh ?

Let's create a client that can do http-calls, especially the POST (task A.)

TeamMemberClient(httpClient: HttpClient)
A. Can call the team-backend and POST a new team member

and inject it in another service which has the responsibility to manipulate data (tasks B, C and D).

InviteTeamMemberService(teamMemberClient: TeamMemberClient, errorService: ErrorService, router : Router)
B. Will show an error message (in global snackbar) if failed
C. Will redirect to Team dashboard if successful
D. Creates FormGroup for team member informations (also stores the data and validates it)

that we inject inside the initial lighter component that does the two last UI features (tasks E and F)

InviteTeamMemberComponent(inviteTeamMemberService: InviteTeamMemberService)
E. Displays the data in the template
F. Keeps track of the loading state of the query

Avoid Input/Output and prefer injecting specialized services

You can sometimes avoid Input/Output drilling (more specialized article) with either service or content injection, this will look like a simple advice to Redux advocates, but here is the good news : we don't need redux !

Here is an example of input drilling where a 2 level deep button opens a modale upper in the hierarchy, all managed by a smart ancestor component !

Valuable code is spread among all the components and useless Input/Output mechanism pollutes the intermediary. On top of that, the ancestor component contains the data, and has the responsibility to update it. Here's what you would prefer :

This was an example of refactoring using service. You can also avoid one layer of Input/Output with content projection. In the example above, you could get rid of @Output() edit if you directly inject the button of the action.

What are the advantages of doing so ?

Synchronization within your team : You can still use old patterns, but don't make it a must ! Just communicate on your rules with your team and write coding guidelines that every one of you has read. You'll see tremendous improvement very quickly
Simpler hierarchy : your DOM is not polluted with useless container components. CSS is simpler, and inspecting the application is simpler too ! You can understand it faster ! If other teams use the same coding guidelines, you'll be able to switch teams easily.
Easier onboarding, as a new developer you know where things are by opening a package : services, components and pipes are doing what you expect them to do ! Also, less wrappers means less clicks to get to meaningful code !

AWS Fargate: harness the power of serverless for long-running computational tasks

Tue, 14 Sep 2021 00:00:00 GMT

TLDR: We all know the advantages of serverless computing: pay only for what you use; scale up and down with ease; abstract away the complexities of managing servers. In this article, I illustrate how you can acquire these benefits for long computational tasks which are too large for Lambda or Google Cloud Functions to handle. I share a case study from a recent project and how we used AWS Fargate (along with S3, Lambda, DynamoDB, and more!) to create a robust task architecture.

Recently, a client approached us with a state-of-the-art modelling system which had been designed by industry-leading domain experts. The techniques used in the modelling software meant that, for its use-case, the software could run simulations in an order of magnitude less time than the competing solutions.

Despite being an industry-leading scientific model, the delivery of that solution through software was holding the product back from its full potential. The software interface was a Python GUI and was distributed as source code directly to users. The model could take hours to run and if it failed (often due to invalid input), the modeller would only discover this error on returning to the program after several hours away, only to correct the error and rerun the test case. Often, multiple variations need to be simulated, however the desktop application couldn't execute multiple runs concurrently (as the software is limited by the processing power of the machine it runs on).

The mission

We wanted to bring this software to the market as an industry-leading product in terms of scientific modelling and software engineering. We would leverage cloud services to mitigate some of the disadvantages of the desktop implementation.

Here are some of the main things we wanted to add to the current functionality of the model:

Improved distribution by migrating to the cloud:
- Easier on-boarding (no software installation - just login).
- Improved security (no user access to source code).
- Run the model from any computer.
Increased scalability:
- Execute multiple model runs concurrently to save users' time.
Improved user experience:
- More data validation rules to prevent failed model runs.
- Email notifications for when the model has completed a run.

The challenge

The abstract problem to solve was that we had a 500,000-line black-box program that was tightly coupled to the environment and file system of the machine it was running on. In our case, three issues stand out:

The model code is tightly coupled to the modeller's development environment - it calls os.system in many places to run shell commands on specific libraries (these included dependencies on GNU Fortran - not something you see in your everyday web application!).
The model code is tightly coupled to the modeller's file system - it creates intermediate output files to save data between different steps in a run. This decision was initially made from a scientific standpoint - a modeller might want to check some intermediate output - and had stuck as a design pattern across the software.
The model code has significant complexity, and re-engineering it came with risks to the validity of the results.

These constraints meant that we needed:

To run the model code on a system with the same environment as the modellers' computers.
A way to persist intermediate files in the cloud between different runs.

The strategy

The high-level flow of a modeller executing a model run would go like this:

The modeller goes to the web application from their browser and configures a new run. Parameters that they configure are saved to DynamoDB and input files they upload are saved in S3.
The modeller clicks the "execute" button for a specific task, which triggers (a Lambda to trigger) a task in Fargate. This spins up a Docker image registered in ECR with the correct environment and source code to run the model. The task name is passed in as an environment variable.
The task pulls the correct input files from S3 and the user-defined parameters from DynamoDB.
The model task is then executed as it would be on a modeller's local computer.
The model syncs the resulting output directory to S3 and EventBridge and SES are used to notify the user of the status of the run via email.

The architect of the project drew a handy architecture diagram to illustrate exactly how we achieved these steps and I'll go through the steps that we took along the way.

Dockerize the tasks: Fargate + ECR + VPC

The first step was to create an environment to run the model; Docker was designed to solve this very problem ("well, it worked when it ran on my computer!").

A new container was defined and the required dependencies for the project were resolved (we used Poetry to manage the Python libraries).

We used Docker volumes for the project so that any intermediate files created within the container are synced with our outer file systems to inspect and validate. A docker-compose.yml file was also written to simplify bringing the container up and down and passing in environment variables from a .env file.

There are several discrete tasks that the model can run, each one generating different output files. For our use-case, we would have a single Docker image, which contains the model code for every type of task - the task to be executed would be defined as an environment variable (a container is designed to be brought up with a single purpose, then taken down once the task is complete).

Now that we know the task we want to execute, we can define the functions we want to run for each task. We define a TASK_DEFINITIONS variable which stores handlers for each task: preprocessor, executor (where we will run the main task), postprocessor, and failure (the model can be temperamental to user input, and we need to handle failure explicitly). I will come back to the pre- and post-processors in the next section.

The executor is simply a function which makes a call to the model source code to execute the desired task. This means we can define an env variable - TASK_NAME - and run the model locally by accessing bash from within the container:

docker-compose up -d
docker-compose exec <image-name> bash
python main.py

Note: my local machine doesn't have the correct dependencies to run the model code, but the container that I run on my local machine does.

The container can now be migrated to the cloud so that the model code can be accessed from anywhere in the world without the need to manually distribute the source code.

The image is registered with AWS Elastic Container Registry (ECR) where the container can be stored and accessed (Model registry in our architecture diagram). A Virtual Private Cloud (VPC) is configured to accept requests to trigger an AWS Fargate task.

Fargate is "serverless compute for containers"; this means that our container isn't running in the cloud when we don't need it - AWS handles the provisioning of resources to "spin up" a container from our image at the moment we want to run a task. This means that we are only paying for what we use whilst still having the potential to scale ad infinitum (we can trigger multiple tasks concurrently and AWS will just spin up more instances for us in the cloud). The one drawback of this architecture is that Fargate requires some time to build the container when we request for our task to be run (this was around a minute for our (large) container).

You can see Fargate (Model task) sitting at the centre of our architecture diagram within the VPC in ECS.

We also set up Cloudwatch to create logs for each run to help us and the modellers debug issues with the model.

Syncing input and output directories: S3

Until now, the model works in the cloud so long as the required input files are present in the file system of the Docker image; however, input files are specific to the context of a run, and if we want to get meaningful output then we need to pass in the correct input files at the point that we execute a task.

For each task, a new instance of the image is spun-up, so any intermediate files are lost when the container is pulled down and the context is lost (this is fine, as we want different executions to remain distinct from each other).

We set up an S3 bucket to store the inputs and outputs of tasks, which means that the files which we expect to persist between different tasks can be synced between permanent storage from task to task.

Each task was saving its output to a different directory, so within our TASK_DEFINITIONS, we define an input_dir, from which we would pull the contents from S3 and a number of output_dirs to which we would push the output of the model. These outputs might be downloaded by the user or used by a different task as input files. This syncing of files happens in the preprocess and postprocess of each task.

Data model: DynamoDB

As well as input files, the model also depends on a range of input parameters that the user can define. In the desktop app, the program gets these input parameters by reading and parsing a local file called INPUT_PARAMETERS.txt.

Each model run entity has a corresponding entity within a DynamoDB table (in practice there are multiple types of entity available for the user to manipulate). The database follows the single-table design as described in The DynamoDB Handbook by Alex DeBrie and uses dynamodb-toolbox to model entities. The user can interact with these entities via the web application to set properties of the run.

We created a corresponding manager in the dockerized model code for each type of entity in the database (e.g. RunManager) which can get the entity and also perform simple updates on text fields (set error messages etc.). The entity is retrieved from DynamoDB which then overrides the variables which were previously being retrieved from INPUT_PARAMETERS.txt, which otherwise describes a generic run.

The model is now no-longer dependent on a local file for input of discrete parameters; but how does the user configure those variables to get a final valid output to their desired parameters?

Web Application: Lambda + Next.js

It's now time to plug the main pieces together.

We developed a Next.js frontend to interface with the remote model. This comes with all the advantages of developing UI using technologies that were designed to handle UI (i.e. not Python!). This meant our input methods were more interactive and our outputs more visual, as we were able to leverage existing libraries.

We added validation rules to the inputs (with warning messages) to prevent the user running the model with parameters that don't make sense. We accept input files as upload fields of several forms, which are pre-processed to validate that the values in the files were within the expected range before uploading to S3.

We built our backend REST API using AWS Lambda. As with Fargate, Lambda offers flexible scaling, has a nice integration for the back-office tool we created in Retool, and we didn't mind the infamous cold starts for our use case.

Another advantage of using Lambda was that we can use AWS CloudFormation (infrastructure as code solution) to provision our infrastructure from a serverless.ts file which defines all the resources required for an environment. If we wanted to create a new environment, running sls deploy <env-name> would provision the resources defined in the file instead of needing to interact with the AWS console.

Notifying model outcomes: EventBridge + SES

Until now, a specific model task can be triggered to run in the cloud where it will sync the relevant input files, pull the correct input parameters, run the task, and sync the output files back to S3. Finally, we need to notify the user of the outcome of the model (success + results/ error + logs).

In the model, we defined a RunEventsManager which could send an event via AWS EventBridge when the postprocess or failure handler was completed in the model. We added a Lambda in our backend which was triggered by an EventBridge event (because we were using CloudFormation, we just needed to define the event that triggers the lambda in serverless.ts).

When an event occurs, the Lambda uses AWS SES client to send an email to the user, which notifies them immediately when they can check their results - there's no need to come back to check that the model hasn't failed every 15 minutes anymore!

Summary

Using the described architecture, we had delivered a significant improvement to the user experience in running scientific modelling tasks by bringing the software to the cloud and affording it all the advantages of a serverless implementation.

Rendering Blender scenes in the cloud with AWS Lambda

Sat, 28 Aug 2021 00:00:00 GMT

You can fairly easily leverage AWS Lambda to render scenes with Blender. It can make sense to use Lambda functions if you need to render a large number of assets in little time and if each asset is simple enough to be loaded and rendered in reasonable time (within the maximum lifetime of a Lambda) given the limited resources offered by Lambda functions (maximum 6 vCPUs and 10GB of RAM at the time of writing). If you have to render more complex assets, going with EC2 instances or AWS Thinkbox Deadline is a better solution.

When I came up with the idea of running Blender on Lambda, I naturally checked if someone was doing it somewhere and found out that @awsgeek had done something similar in 2019 so at least I knew it was possible. What we are going to present in this article slightly differs from his approach though.

The architecture

Here is the very simple architecture that allows you to render Blender scenes on AWS Lambda stored on an S3 Bucket. The final renders are stored into the same bucket.

The files you will need to run this "at home" (in the cloud)

To run this on your own AWS account, you will need four files:

Dockerfile describes the container image that each lambda function execution is going to run
serverless.yml describes the infrastructure above
app.py handles the event from the Lambda infrastructure and starts up blender
script.py manipulates Blender from the inside to lad a scene and render it.

File structure

Here is how these files should be arranged:

project directory
+-- Dockerfile
+-- serverless.yml
+-- app
|   +-- app.py
|   +-- script.py

Dockerfile

For the Dockerfile, I started from a great set of Docker images for Blender maintained by the R&D team of the New York Times. We're using the latest image that runs Blender rendering with CPU only (as Lambdas don't have GPU capabilities unfortunately).

FROM nytimes/blender:2.93-cpu-ubuntu18.04

ARG FUNCTION_DIR="/home/app/"
RUN mkdir -p ${FUNCTION_DIR}
COPY app/* ${FUNCTION_DIR}
RUN pip install boto3
RUN pip install awslambdaric --target ${FUNCTION_DIR}

WORKDIR ${FUNCTION_DIR}

ENTRYPOINT [ "/bin/2.93/python/bin/python3.9", "-m", "awslambdaric" ]

CMD [ "app.handler" ]

Serverless configuration file

The serverless.yml file is fairly simple:

service: blender-on-lambda
frameworkVersion: '2'
configValidationMode: error

custom:
  destinationBucket: blender-on-lambda-bucket

provider:
  name: aws
  region: us-east-1
  memorySize: 10240
  timeout: 900
  ecr:
    images:
      blender-container-image:
        path: ./

  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "s3:PutObject"
        - "s3:GetObject"
      Resource:
        Fn::Join:
          - ""
          - - "arn:aws:s3:::"
            - "${self:custom.destinationBucket}"
            - "/*"

functions:
  render:
    image:
      name: blender-container-image
    maximumRetryAttempts: 0

resources:
  Resources:
    S3BucketOutputs:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: ${self:custom.destinationBucket}

In this example, I am using the largest available Lambda functions with 10GB or RAM and 6 vCPUs so that the rendering goes as fast as possible.

The Python Lambda handler

The app.py file contains the logic that will be triggered by the Lambda infrastructure and it will receive the event passed as a trigger. In this case, the event must contain the width and height of the image to be rendered.

import json
import os


def handler(event, context):
    os.system(f"blender -b -P script.py -- {event.get('width', 0)} {event.get('height', 0)}")
    return {
        "statusCode": 200,
        "body": json.dumps({"message": 'ok'})
    }

The Python Blender script

And finally, here is the python script that will be executed from inside Blender to do the actual rendering.

from datetime import datetime
import os
import sys

import boto3
import bpy


argv = sys.argv
argv = argv[argv.index("--") + 1:]

s3 = boto3.resource("s3")
BUCKET_NAME = "jrb-material-renderer-bucket"
filename = f"{datetime.now().strftime('%Y_%m_%d-%I:%M:%S_%p')}.png"

s3.Bucket(BUCKET_NAME).download_file("scenes/scene.blend", "/tmp/scene.blend")

bpy.ops.wm.open_mainfile(filepath="/tmp/scene.blend", load_ui=False)
bpy.context.scene.render.filepath = f"/tmp/{filename}"
bpy.context.scene.render.resolution_x = int(argv[0])
bpy.context.scene.render.resolution_y = int(argv[1])
bpy.ops.render.render(write_still = True)

s3.Bucket(BUCKET_NAME).upload_file(f"/tmp/{filename}", f"renders/{filename}")

Cost

In the us-east-1 region of AWS, the largest available lambda will cost you $0.01 per minute. The request charge (the amount charged for every invocation) is probably negligible in comparison. Also, bear in mind that the generous free tier of Lambda might cover your costs! Same for S3.

I hope this can help anyone facing the same need that I did.

If you're interested in offloading your Blender render jobs to the cloud, follow me on Twitter. I'll be posting more content in the coming weeks such as:

cost comparison with using other cloud providers
the cost impact of using GPU-enabled EC2 instances instead of Lambda
leveraging AWS EFS instead of S3 to load the scene into Blender

If there are other topics you'd be interested in, let me know on Twitter! My DMs are open.

Sending Financial Data to Salesforce Without the Hassle Thanks to Codat and Amazon AppFlow

Wed, 11 Aug 2021 00:00:00 GMT

We’ve recently built a secure data pipeline for a customer called Founders First Capital Partners. They needed to automatically load large amounts of financial data into Salesforce, their CRM, in order to make investment decisions. We used Amazon’s managed data integration service — AppFlow — which takes care of creating a secure connection and sending data to Salesforce. To retrieve the data securely from several financial tools, we leveraged Codat. Codat's single API powers robust integrations to over 29 different financial tools including QuickBooks, Stripe, Plaid and Oracle NetSuite. Codat takes care of all the heavy lifting involved in these integrations — authorization, data standardization,ongoing connectivity — leaving us free to focus on how to best use this data.

The architecture

We designed a simple serverless architecture that fit our needs.

Benefits of using these tools

Why would you do this instead of “simply” writing the connection to the APIs?

Less code to write. All you have to do is write code that puts data in an S3 bucket and AppFlow will take care of sending it to Salesforce (or any other supported SaaS tool you choose).
All the usual benefits of serverless and managed services: you only have to focus on the business logic that matters to your use case. AWS ensures your solution is always up and running (with an SLA of 99.9%). There are no software updates or patching to worry about. It scales with your needs.
Enhanced security: for each SaaS that AppFlow connects to, AWS establishes a secure AWS PrivateLink which is a great security guarantee without any additional effort.
The standardization of data provided by Codat allowed us to manipulate data from various sources into one consistent data structure, which is a huge time-saver.

Trade-offs of our solution

AppFlow can sometimes throw error messages that are not very helpful, which can be frustrating. But, just like with other AWS products, their team is open to feedback and continuously improves AppFlow.
The community adoption of AppFlow still seems to be nascent (based on the number of Stack Overflow discussions on the topic), which is a bit of a shame given how helpful this service is. As a result, it can be a struggle to find resources to help you get up and running.

The combination of Codat with Amazon AppFlow allowed us to load large amounts of critical financial data into Salesforce in a very short amount of time, with minimal development effort. I hope you can also save time by using these amazing tools!

Thanks to Nate Ritter, CTO of Founders First Capital Partners, for giving us the opportunity to work on this small piece of serverless sorcery. Founders First Capital Partners are based in San Diego, CA and support diverse founder-led small businesses and those in low to moderate income areas by providing them with funding and acceleration services.

Hasura: GraphQL Without the Baggage.

Wed, 04 Aug 2021 00:00:00 GMT

Spoiler Alert: Star Wars references below. Proceed at your own risk...

The introduction of GraphQL back in 2015 garnered a lot of hype in the development sphere - allowing clients to describe exactly what data they require (preventing over-fetching unneeded information) and unifying multiple resources into a singular request. This substantially improved scalability and reusability of backend endpoints, and makes the development experience all-in-all more pleasant.

So you might be thinking to yourself: "Great. What's the catch?"

Well... Implementing GraphQL into your backend will inadvertently increase complexity - you'll need to dedicate time on defining types, efficient resolvers, mutators, queries, etc. If you're working as a full-stack developer, you might find that using GraphQL is kind of like taking a dollar from one pocket and putting it in another. The added complexity can start to outweigh the benefits, especially on smaller scale projects. On top of that, caching is complex with GraphQL. With no in-built caching capabilities, this often becomes a pain point for developers.

That's not to diminish or underplay the clear benefits of using GraphQL.

However, I want to use this opportunity to touch on some of the less talked about vices of GraphQL, and present Hasura as a possible solution to some of these concerns. Hasura is a GraphQL service that automatically generates GraphQL endpoints from your database schemas, taking out a lot of the complexities and manual work needed. Simply by connecting Hasura to your SQL database, it will generate schemas and resolvers using the existing tables and views. It's quite magical in how quick it is to get up and running...

GraphQL Complexities

Boilerplate & Initial Setup

As you'll usually find with most other large scale dependencies or libraries, the addition of GraphQL will require some level of initial setup and boilerplate, you're adding a new layer to both your frontend and your backend. You're probably going to need to add a GraphQL client, define the types, schemas, error handling, caching on the frontend level. On the backend side, you'll need to define multiple resolvers to fetch different views of the data (get all the data/get a single datapoint with an id), as well as defining mutations for creating new datapoints, editing existing data, and deleting data.

Now you could argue that these would all still be required if you used normal REST endpoints - and you would be right. However, on top of these, you'll need to deal with typing on the backend, caching (you can't rely on native HTTP caching like REST APIs), higher order components, and additional validators to catch some edge-cases that arise from defined schemas.

A solution commonly taken on the backend side, is to define boilerplate abstractions to take the repetition out of having to define basic resolvers/queries for new data classes you add into your application, but again this takes time to setup (plus you're adding more complexity). While this might not be an issue for larger projects that can reap the benefits of using GraphQL, it becomes a massive swaying point away for small projects.

Resolver Hell

Resolvers are the functions that resolve the values for any field in a schema. They can return primitive values, or objects that get further broken down into more primitive types. You'll find that these typically return values from databases or other APIs.

Now there are a number of databases and ORMs that come with GraphQL mappers that will automatically handle resolvers for you when you are defining your data in objects. For example, in a project that I worked on, we use GORM, which handles resolving and schema generation. However, these do come with their overheads, and you would need to define your data models according to these data access tools.

In many cases you'll find that you're dealing with your database directly, and you'll be doing the DB querying yourself. This opens a whole new can of worms to deal with.

A common issue that many face is over-querying the database. Let's demonstrate this with an example:

You are creating an application to manage a large movie cinema company with multiple branches. You have a query that fetches a list of all the cinemas, the screens in each, the movies showing on each screen, and which production companies released them.

Now presume that there are 5 cinemas (branches), with 10 screens each, each playing 5 movies at any given point. This means that we have 1 query to get the cinemas, 5 queries for each cinema getting the screens, and 50 queries for the movies playing on each screen (50 calls to get the details of the movie playing on each screens). That comes down to a total of 56 database calls for this query.

Say you wanted to fetch some basic metadata about each of these movies as well (such as the production companies), that would be another 50 database calls added, which would increase our total to a whopping 106 database calls...

Cinema example above visualised.

This can be mitigated by strategising and coming up with efficient queries and resolvers, but the problem remains that it is very easy to fall into these common patterns that will result in overfetching. It's just the price that comes with allowing the client to specify their data needs and schemas.

There are solutions to deal with this - for example, Dataloader will batch some of these database calls together and cache some of the results. However, these solutions are far from perfect and they have limitations when making certain queries. Not to mention that by adding new dependencies, we start inadvertently increasing complexity of the application.

Performance & Caching

With complex views on the data, comes the cost of query complexity and the number of database calls increasing. It becomes a balancing game... You'll want to use GraphQL's ability to define graph-based traversals to fetch multiple resources (in fact, this is probably the single biggest selling point IMHO), but use it too much (or irresponsibly), and you might find that your queries become slow and start to massively affect performance.

Take the Star Wars example that is shown on the homepage of the GraphQL website:

type Query {
  hero: Character
}

type Character {
  name: String
  friends: [Character]
  homeWorld: Planet
}

type Planet {
  name: String
  climate: String
}

Let's try and visualise it with a flowchart, with the main trio of the original series. If Luke Skywalker is the hero of the original series (which is a false statement, because we all know the Ewoks were the true heroes), he has several friends, and each of these friends have a homeWorld.

GraphQL allows you to traverse the graph of Characters by defining resolvers to fetch the friends. Now this is a fairly simple example, and you'll find that real world data will usually have more complex relations and hence this graph will become larger. This will take a toll on performance, and you may find that certain queries will take unreasonably long to run.

A common approach taken to solve this problem is through caching, and although it won't work well for frequently changing data, this can help reduce the number of queries that need to be made regularly on data that rarely changes.

As an example, Apollo can allow you to define and control caching structures on your web applications, but if you have multiple frontend applications (say a web, Android, and iOS app), you'll probably need to handle caching on each of these levels separately. Again, added complexity that will be introduced by using GraphQL.

However, this options is only viable for data that isn't changing regularly... So in the case where Leia's home planet of Alderaan gets obliterated by the Death Star, or Han Solo and Leia become separated (ergo are no longer considered friends) due to their son falling to the dark side of the force, you may find that caching will result in data inaccuracies. This is where the importance of having well-written and efficient resolvers and database calls comes into play yet again.

And with that, all may seem lost, but from the darkness of caching, boilerplate, and complex resolvers comes the saving grace of Hasura...

Hasura

Hasura is an open source service that allows you to take your relational databases, and automatically generate GraphQL APIs on top of them. Effectively, this removes one of the biggest challenges that backend developers using GraphQL face, which is creating efficient SQL queries and resolvers.

Looking at the challenges posed above, the issue of excessive boilerplate code & high initial costs effectively become neutralised by using a tool like Hasura. I had a Postgres database running on my machine, and it took me a total of 15 minutes to spin up Hasura with Docker. Just by changing the docker-compose, I pointed it to my database, and it loaded up already connected and hosting a GraphQL endpoint with my defined views! Honestly, it felt like some form of vodoo magic! The steps I followed were really quite simple:

Fetch the docker-compose file with curl.
Run docker-compose up -d
Get yourself a cup of coffee while you wait for it to run.
Open the console page at localhost:8080/console
Click on the data tab, and click the "Connect Database" button.
Add a name for your database, the SQL flavour, and lastly the database URL.
Click the connect button.
After it's done connecting, click on the "GraphiQL" tab, and see that your database tables can be queried out of the box!

Now, a common question that may come up about the initial cost of using a technology is how well it integrates with existing codebases. Presume that you're already using a GraphQL endpoint for the backend of your application, and you want to incrementally introduce Hasura to handle the basic queries and subscriptions (yes, it comes prebuilt with subscriptions, and that can be an absolute pain with certain GraphQL servers). You can use a feature called Remote Schemas to point Hasura to your already existing GraphQL endpoint, and it'll automatically combine the schemas together to create a unified GraphQL endpoint that your frontend can use. Additionally, you can use this to delegate more complex business logic in your code to other services (such as a payment API).

The second challenge touched on above was the issue of inefficient and complex resolvers. Without getting into much detail, Hasura compiles down GraphQL queries using abstract syntax trees, validates and checks permissioning at this level, and then constructs the entire query into one raw SQL query. On top of this, they do 2 levels of caching (one at the SQL compilation level, and another at the database level with prepared statements) within their architecture to reduce the amount of times these compilation steps are required. They're building and improving on this architecture and performance with every update. For example, they've introduced JSON aggregations on the database level, effectively removing the need to flatten the results from the database calls in the backend - they claim this increases performance anywhere from 5-8 times! For those who are interested, they have an interesting article on their website talking about their architecture in more detail.

Hasura's Data Wrapper (Source: hasura.io)

Lastly, we touched on the issue of caching & performance. Within the data wrapper layer, there's already 2 levels of caching that will significantly improve performance. Additionally, there is another layer of caching, which is on the query level, that effectively allows you to cache the response being sent back to the client for a defined period of time. You can make a call with the @cached directive to cache the response, supplying an optional ttl argument (how long to cache the response):

query MyCachedQuery @cached(ttl: 120) {
  users {
    id
    name
  }
}

This caching comes working out of the box. In contrast to common caching approaches (such as using Apollo), having this caching done on the server side can be beneficial for unifying the responses on any given platform at any point of time. Imagine you have a cached query for the top players in a leaderboard. If the caches are set at different points on the client, there may be inconsistencies in how this data is displayed. Caching on the server-side circumvents this.

On top of that, Hasura's performance isn't only due to compiled queries, efficient resolvers and caching. There's a ton more work put into improving speed and performance and it is a regularly improved and maintained project that's already heavily utilised in the industry.

So to demonstrate the Star Wars example we saw above, I went ahead and created a database schema to have the following tables:

Character
Planet
friendsWith

I did all of this through the Hasura data admin panel (sort of like a lite version of PGAdmin). Through this data panel, you can choose which tables/views can be tracked by the GraphQL schema, and you can define table relationships and how they manifest in the GraphQL schema.

Switching to the "GraphiQL" tab, we can now see that the explorer now has a bunch of different datapoints:

You can fetch the list of each of the data types by calling the basic data type (for example, Planet), and this comes with pagination and filtering out of the box. Alternatively, if you're looking for some form of aggregation on the data, you can use the _aggregate datapoint, which can run some basic aggregation on the data (such as averaging a field, counting, finding the min/max the field, or even getting the standard deviation). Lastly, you can get a single data entry by using the by_pk variation.

Now, to see this in action, we construct a query asking for the first character in our character table. We ask for the name of the character, their home planet (along with its associated climate), and a list of their friends.

To be able to do this in the matter of a couple minutes is something that I still find magical.

I'll end here with some of the selling points that I think make Hasura really shine:

Automatically generates the schemas and resolvers for your GraphQL endpoints (less pain).
Comes prebuilt with option variables that you can use in your queries (for example, limit and offset, which will allow you to more or less handle lazy loading of datapoints).
Currently works with a number of SQL databases (Postgres, Microsoft SQL, Amazon Aurora, and Google Big Query) with more coming soon.
Easy to integrate with authentication and permissioning platforms, such as Auth0.
Gives you a web UI to manage the GraphQL endpoint, as well as an interface to manage the database and its views itself (think PGAdmin).
Using Remote Schemas, you can hook Hasura up with other GraphQL endpoints and other services to unify your endpoints.
Using Actions, you'll be able to define calls to custom handlers that will execute more complex business logic (which opens the door to serverless by using Lambdas)
You can define event triggers to call actions automatically (when a user signs up, send an email via a Lambda function).
Highly maintained and used by the community, with over 23k stars on Github and 250m+ downloads.

Conclusion

GraphQL provides some fascinating functionality that aims to make the data layer of applications unified and flexible. As with most things, this comes at a cost. The initial setup and boilerplates will mean that you will need to spend time upfront in creating the right infrastructure for using GraphQL. On top of that, the flexibility in schema means that you will need to have additional considerations when it comes to performance, and you will need to come up with efficient queries and resolvers. Lastly, with GraphQL we lose the native HTTP caching capabilities, meaning that caching is not standardized and becomes cumbersome additional work.

Hasura, acting as a service layer between your databases and clients, can automatically generate performant GraphQL queries, mutators, and even subscriptions, with minimal effort on the your end.

Whilst Hasura's solution to automatic generation of schemas and endpoints might not offer the full flexibility that large-scale highly complex applications might need, it opens the door to using GraphQL on smaller scale projects where the startup costs of introducing it would outweigh the benefits. I believe this is where Hasura will excel, and where it will ultimately find its place in the GraphQL market.

Writing a native Ionic plugin for Capacitor in less than 30 minutes

Mon, 12 Jul 2021 00:00:00 GMT

TLDR: When the pandemic first started, I decided to develop a contact-tracing mobile app. I was studying Computer Science at the time and decided to use the same cross-platform framework that we were using on our course - Ionic. I wanted to use Google's Nearby Messages API to share packets of information between iOS and Android devices via Bluetooth, but I couldn't find a plugin for this, so I decided to write my own!

View commits: Android plugin → iOS plugin → Import native plugins → Implementation

Ionic is a cross-platform mobile framework which allows you to develop an app using JavaScript/HTML/CSS and share this single implementation across different native devices. This Ionic project can then be compiled to native source code (Android/iOS) using Ionic's tool, Capacitor.

Using a cross-platform framework like this to develop your mobile app is almost always a good idea, because it means you can maintain a single codebase which runs on both Android and iOS. This means you're not duplicating business logic across two different codebases in two different languages with two different teams (to be avoided).

If you're looking to write some native code in a cross-platform project but you're still deciding which framework to use, check out this Theodo blog post on React Native as well!

Step 1 - Search for existing solutions

Firstly, let's have a quick check that one of the default plugins doesn't do the job for us already.

Secondly, let's search through the community plugins to find the feature we're looking for.

No luck finding an existing plugin? Don't worry - let's move on to step 2!

Step 2 - Start a new app with Ionic + Capacitor

I'm starting a new project from scratch just to show you how it's done.

npm install -g @ionic/cli

ionic start <app-name>
cd <app-name>
ionic serve <-- you can serve your project on localhost

The /src directory of our Ionic project holds the source code for our cross-platform app.

We can use Capacitor to compile this into two native projects:

npm install @capacitor/cli @capacitor/core

npx cap init
npx cap add ios
npx cap add android

When we make changes to our /src directory, we can apply those changes to our native projects by running npx ionic build && npx cap copy.

We now have our project set up and ready to add a custom plugin.

Step 3 - Check the documentation

Capacitor is well-documented and it's worth linking some references here to accompany this guide:

Getting started with Android

Android reference

Getting started with iOS

iOS reference

Step 4 - Define a strategy

There are three main steps to writing a plugin:

(iOS & Android) Write a class in each native project to hold the native code we want to run (this is our plugin).
(iOS & Android) Register these classes inside the bridge of our native projects to expose the methods.
(Javascript - Ionic) Import and call the methods of our registered plugin.

We should avoid duplicating code as much as much as possible (DRY), so all your business logic stays written in JavaScript, keeping the bare minimum written natively.

Your plugins should be narrow in scope (do one specific thing) - make sure to follow this philosophy and give your plugin a specific name ('Plugin' is a bad name!).

We have two main strategies for calling our plugins:

An (asynchronous) method: we call some native method which we can await in our business logic in Ionic.
Event listeners: define a listener in our business logic. Events can be triggered in our native code to pass payloads back to the business logic.

If we are subscribing to a stream of events (expecting our method to resolve with multiple payloads over a longer time period e.g. geolocation/ Bluetooth updates), then event listeners are the appropriate way to handle our native calls.

Step 5 - Write and register the Android plugin

[View the commit]

Open the /android directory of your project in Android Studio.

In /app/src/main/java/../../.. (next to MainActivity.java), we can create a new file (I'm calling mine IonicNativePluginExample.java) and write a class which extends the Capacitor Plugin class:

package /* <package-name> */; // e.g. io.ionic.starter

import com.getcapacitor.JSObject;
import com.getcapacitor.Plugin;
import com.getcapacitor.PluginCall;
import com.getcapacitor.PluginMethod;
import com.getcapacitor.annotation.CapacitorPlugin;

@CapacitorPlugin(name = "IonicNativePluginExample")
public class IonicNativePluginExample extends Plugin {

    @PluginMethod
    public void NativeMethod(PluginCall call){
        JSObject result = new JSObject();
        result.put("message", "Hello Android user!");
        call.resolve(result);
    }

    @PluginMethod
    public void NotifyListeners(PluginCall call){
        JSObject result = new JSObject();
        result.put("message", "Hello Android user!");
        notifyListeners("EVENT_LISTENER_NAME", result);
    }

}

Next, we can register this plugin in MainActivity.java where Capacitor initialises its bridge:

package /* <package-name> */; // e.g. io.ionic.starter

import android.os.Bundle;
import com.getcapacitor.BridgeActivity;

public class MainActivity extends BridgeActivity {
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        registerPlugin(IonicNativePluginExample.class);

    }
}

We can run our project like any other native Android project, by pressing the 'play' button in Android Studio.

If you're looking to develop in Kotlin - Android Studio comes with tooling to convert this Java class to Kotlin (right-click the file you want to convert).

Note: you might need to edit the Gradle distributionUrl to get the project build to succeed - there's an example in my repo here.

Step 6 - Write and register the iOS plugin

[View the commit]

Open the ios/App directory of your project in Xcode.

In /App (next to AppDelegate.swift), we can right-click on the directory to create a new swift file (I'm calling mine IonicNativePluginExample.swift). This will hold our plugin class with our IonicNativePluginExample method.

import Foundation
import Capacitor

@objc(IonicNativePluginExample)
public class IonicNativePluginExample: CAPPlugin {

	@objc func NativeMethod(_ call: CAPPluginCall) {
		call.resolve(["message": "Hello iOS user!"])
	}

	@objc func NotifyListeners(_ call: CAPPluginCall) {
		self.notifyListeners(
			"EVENT_LISTENER_NAME",
			data: ["message": "Hello iOS user!"]
		)
	}

}

Next, we must register our plugin in a new Objective-C file called IonicNativePluginExample.m (same name as our plugin file but with a .m extension). When prompted by Xcode, create a Bridging Header file (which is an empty file called App-Bridging-Header.h), then register the plugin like so:

#import <Foundation/Foundation.h>
#import <Capacitor/Capacitor.h>

CAP_PLUGIN(IonicNativePluginExample, "IonicNativePluginExample",
           CAP_PLUGIN_METHOD(NativeMethod, CAPPluginReturnPromise);
           CAP_PLUGIN_METHOD(NotifyListeners, CAPPluginReturnPromise);
)

We can run our project like any other native iOS project, by pressing the 'play' button in Xcode.

Step 7 - Import and call our plugin in Ionic

[View the commit]

Finally, we can now access the plugins we have written from our Ionic project. I've created a /plugins directory to export mine from. We import registerPlugin from Capacitor to find the plugin we have registered in Android and iOS:

import { registerPlugin, Plugin } from "@capacitor/core";

// we can take advantage of TypeScript here!
interface NativePluginInterface extends Plugin {
  NativeMethod: () => Promise<Record<"message", string>>;
  NotifyListeners: () => Promise<void>;
};

// it's important that both Android and iOS plugins have the same name
export const IonicNativePluginExample = registerPlugin<NativePluginInterface>(
  "IonicNativePluginExample"
);

Now we can call the methods in our plugin and access the native code:

import { IonicNativePluginExample } from './plugins/IonicNativePluginExample'

...

// add a listener to native events which invokes some callback
IonicNativePluginExample.addListener("EVENT_LISTENER_NAME", ({ message }) =>
    console.log(message);
);

// destructure the methods to call our native code from our non-native app
const { NativeMethod, NotifyListeners } = IonicNativePluginExample;

// native methods are asynchronous
const { message } = await NativeMethod();

// this method will trigger our event listener
NotifyListeners();

Note: to see our changes reflected when we run the project natively, we need to run npx ionic build && npx cap copy.

We now have a cross-platform app set up ready to write a native implementation. We run the same business logic from our Ionic app and get a different implementation for each device we run on.

I made a small edit to our Ionic project to create the simple implementation below - you can view the commit here.

More complex plugins

With the above setup, you should be ready to write more complex native code to suit your needs for a cross-platform Ionic application.

I wrote a native Plugin for Google's Nearby Messages API, which is a publish-subscribe API made by Google which facilitates the transfer of information between internet-connected Android and iOS devices.

Nearby Messages uses a combination of Bluetooth, Bluetooth Low Energy, Wi-Fi, and near-ultrasonic communication between nearby devices to create a unique pairing. This pairing is then used to send small payloads over the internet between nearby devices.

Here's a quick code snippet from the Android plugin:

...

import com.google.android.gms.nearby.Nearby;
import com.google.android.gms.nearby.messages.Message;
import com.google.android.gms.nearby.messages.MessageListener;

...

@NativePlugin()
public class NearbyMessagesPlugin extends Plugin {

		private Message mMessage;
    private MessageListener mMessageListener = new MessageListener() {
        @Override
        public void onFound(Message message) {
            JSObject result = new JSObject();
            result.put("message", new String (message.getContent()));
            notifyListeners("FOUND_MESSAGE", result);
        }

        @Override
        public void onLost(Message message) {
            JSObject result = new JSObject();
            result.put("message",  new String (message.getContent()));
            notifyListeners("LOST_MESSAGE", result);
        }
    };

    @PluginMethod
    public void Subscribe(PluginCall call){
        Nearby.getMessagesClient(getContext()).subscribe(mMessageListener);
    }

    @PluginMethod
    public void Publish(PluginCall call){
        String value = call.getString("message", "-");
        mMessage = new Message(value.getBytes());
        Nearby.getMessagesClient(getContext()).publish(mMessage);
    }
}

Event listeners are appropriate here because although we don't know when/if the phone will pick up a Bluetooth signal, a side-effect can still be triggered in the business logic when it does. Subscribe can be invoked to subscribe to mMessageListener and listen for Bluetooth messages (or we could just call Subscribe in the onCreate of our native code if we wanted to).

Summary

Once you've got a simple implementation, you can move on and write some more complex native code.

If you want a quick-start, you can fork my repository on GitHub, but following along with the article and doing it yourself will really help your understanding.

RisXSS, the missing ESLint rule for React and Vue

Mon, 05 Jul 2021 00:00:00 GMT

XSS attacks are expected to be ranked 3rd in the OWASP Top 10, compared to their 7th rank in 2021 (source). The pace of application creations is increasing and so is the surface of attacks, so making your applications secure is more and more challenging and needed. React and Vue are some of the most used frameworks to build applications and include built-in protection for some types of XSS by escaping user input by default. However, there may be some cases when they need to display user content as raw HTML (a blog post with style for instance).

export const BlogPost = ({ post }) => {
  return (
    <div dangerouslySetInnerHTML={post} />
  );
};

(All the examples in this article use React but are similar with Vue).

A first attempt to prevent XSS attacks

This opens a large hole in the XSS protection. A secure solution is to first sanitize the content with a library that will remove unsafe parts of the content while keeping useful tags and attributes (for styling purpose for instance). The most popular libraries for this are DOMPurify and js-xss.

This solution is not always known though. On a website I worked on, I saw the use of dangerouslySetInnerHTML that was not properly sanitized and introduced a real XSS vulnerability. My first reaction back then was: "How can I make sure this will never happen again?" So I searched the Internet and I happily found ESLint rules designed precisely to prevent the use of dangerouslySetInnerHTML and v-html. I added these rules on my projects and some time later, again, I found another XSS vulnerability.

How did it happen? Easy: some developer wanted to display sanitized content with style, so they used v-html while sanitizing the output first and added a comment to disable the ESLint rule for this line. This is correct in this case. Then, another developer copied-pasted (who does not?) this code in another page, but the content was not sanitized.

When you have no other means than to disable a rule to do something, disabling the rule becomes the norm, and the rule loses its purpose.

Custom ESLint rule to the rescue

This is when I talked with some colleagues and started a custom ESLint rule to warn about insecure HTML only if the HTML is not sanitized. Corentin Normand and Guillaume Klaus implemented the logic (and even wrote a tutorial about it) and it became RisXSS.

RisXSS warns you about insecure use of dangerouslySetInnerHTML and v-html but only if the content is not sanitized. The following snippet will not display a warning:

import { sanitize } from 'dompurify';

export const BlogPost = ({ post }) => {
  return (
    <div dangerouslySetInnerHTML={{ __html: sanitize(post) }} />
  );
};

By offering a smart ESLint rule, RisXSS achieves the best of both worlds:

preventing XSS
allowing developers to display content with style

Now that you understand the need of RisXSS, go check out how to install it on your project on the RisXSS homepage.

How to Choose Between Different Code Reusable Pattern in Vue.js?

Mon, 28 Jun 2021 00:00:00 GMT

As developers at Theodo, we are always looking for efficient ways to reuse our code without repeating ourselves (DRY principle). In the context of an e-commerce project using the Vue Storefront framework and Vue 2, we had the opportunity to study the best ways to reuse functional behaviors in Vue.

We found a lot of solutions in the Vue documentation such as Mixins, composition API and scoped slots. I hope this article will help you choose yours according to your projects.

Here are the key questions I will answer thereafter:

What is the best method to share logic among code in Vue 2?
What are the advantages and drawbacks of using the newly developed composition API, scoped slots, or Mixins to reuse some logic?
When should you use one over the other?

TL;DR:

Using Mixins is one of the ways to manage shareable logic in Vue. But they have many drawbacks and should be avoided in both Vue 2 and Vue 3.
Scoped slots are a great alternative to Mixins for Vue 2 and they still have their place in Vue 3.
The Vue 3 composition API is a powerful tool to create readable component shareable logic. It is the option I would recommend to use in most cases but is for now limited to Vue 3.

Context

We had the responsibility to develop different authentication connections for a web platform (Google & Facebook OAuth in addition to a basic email/password authentication).

In short, we needed to create three buttons that trigger three different actions. Our buttons looked very similar so we wanted to avoid duplicating the UI code. We thus created a single LoginButton.vue taking 2 props: label and icon. The button triggers an event when clicked (event named click).

// components/LoginButton.vue

<template>
  <button @click="$emit('click')">
    <img v-if="icon" :src="icon.src" alt=""/>
    <span>{{ label }}</span>
  </button>
</template>

<script>
export default {
  props: {
    icon: {
      type: Object,
    },
    label: {
      type: String,
      required: true
    }
  }
}
</script>

Our objective was to:

build the three authentication processes using this generic login button.
organize our three different login logics in a way that would allow them to be used anywhere in the app without repeating ourselves.
keep the code as readable as possible

Before even starting, we wondered if creating basic js functions for each connection methods in a separate JS file would be enough. The issue is that the implementation of an oauth connection requires access to the component lifecycle hooks (the initialization of the oauth client has to be triggered in the mounted hook). This access to the component options thus became a new requirement in our quest for the best solution.

First lead: Define the different connection methods within a wrapping component

This was our first idea. We created a login page component (LoginPage.vue) which uses our login button and defines three methods within it: googleLogin, facebookLogin, and emailLogin. These methods are called when a button triggers its click event.

<!-- LoginPage.vue -->

<template>
    <div class="login-page">
        <button-login label="Login with Google" :icon="googleIcon" @click="loginWithGoogle"/>
        <button-login label="Login with Facebook" :icon="facebookIcon" @click="loginWithFacebook"/>
        <button-login label="Login with email" :icon="emailIcon" @click="loginWithEmail"/>
    </div>
</template>

<script>
import ButtonLogin from './components/ButtonLogin.vue'

export default {
  name: 'LoginPage',
  components: {
    ButtonLogin
  },
  data () {
    return {
      googleIcon: {src: 'path/to/google-icon.png'},
      facebookIcon: {src: 'path/to/facebook-icon.png'},
      emailIcon: {src: 'path/to/email-icon.png'},
    }
  },
  methods: {
    loginWithGoogle () { /* google login logic */ },
    loginWithFacebook () { /* facebook login logic */ },
    loginWithEmail () { /* facebook login logic */ }
  }
}
</script>

This solution had nevertheless many drawbacks as our three login logics are contained in one single component. Consequently:

the LoginPage component is big and unclear
each connection methods is not reusable in other parts of the platform

By looking at the vue docs, we heard about Mixins. At first sight, we believed it was a great solution for the reusability of our login logics.

Second lead: Define each login logic in a different Mixin

A Mixin is an object containing component options (data, computed, methods...) that can be "merged" with a Vue component. During this process, the Mixin options are mixed with the component's options.

How does a Mixin look?

In our case, here was the GoogleLogin Mixin skeleton:

// mixins/GoogleLogin.js

export const GoogleLogin = {
  name: 'GoogleLogin',
  mounted () {
    /* init gapi client */
  },
  methods: {
    loginWithGoogle () {
      /* logic to open google popup, retrieve the authorization code and send it to backend */
    },
    onFailure () {
      /* logic when the sign in method fails */
    },
    onSuccess () {
      /* Logic sending the authorization code to my backend */
    }
  }
}

and our LoginPage:

// LoginPage.vue

<template>
  <div class="login-page">
    <button-login label="Login with Google" :icon="googleIcon" @click="loginWithGoogle"/>
    <button-login label="Login with Facebook" :icon="facebookIcon" @click="loginWithFacebook"/>
    <button-login label="Login with email" :icon="emailIcon" @click="loginWithEmail"/>
  </div>
</template>

<script>
import ButtonLogin from './components/ButtonLogin.vue'
import { GoogleLogin } from './mixins/GoogleLogin.ts'
import { FacebookLogin } from './mixins/FacebookLogin.ts'
import { EmailLogin } from './mixins/EmailLogin.ts'
export default {
  name: 'App',
  components: {
    ButtonLogin
  },
  mixins: [GoogleLogin, FacebookLogin, EmailLogin],
  data () {
    return {
      googleIcon: { src: 'path/to/google-icon.png', alt: 'google icon' },
      facebookIcon: { src: 'path/to/facebook-icon.png', alt: 'facebook icon' },
      emailIcon: { src: 'path/to/email-icon.png', alt: 'email icon' },
    }
  },
}
</script>

Mixins looked like an efficient way to encapsulate and share the different login logics through our application. Nevertheless, after a few tests, we experienced some strange behaviors while using Mixins:

implicit dependencies: the Mixin's methods and data are not defined within the LoginPage component. As such it is confusing to use "hypothetical" methods or data (and to not know where each method or data is defined).
naming collision: when merging the Mixins inside the page component, some of our methods were not called. They were overridden by methods with the same name but defined in another Mixin. As an example, if the FacebookLogin and GoogleLogin Mixins have an onFailure method, only the one defined in FacebookLogin will be called. Indeed, the FacebookLogin Mixin is imported after the GoogleLogin Mixin. In addition, if an onFailure method is also defined in the PageLogin component, then calling this.onFailure will trigger the onFailure method defined inside PageLogin. Consequently, none of GoogleLogin and FacebookLogin onFailure methods will be called (and this will happen without any warning or errors in the console).

We added to the list of our criteria the prevention of implicit dependencies and naming collision.

Knowing that, we were determined not to use Mixins. We decided to find a cleaner way to meet our objectives. It was at that moment we heard about the new composition API of Vue 3.

Third lead : Use the Vue 3 composition API

On September 18th, 2020, Vue 3 was officially released with a new great feature for reusability: the composition API. We won't get into the details of how the composition API works, but in a few words, this API is inspired by React Hooks*. It enables encapsulating component logic into small fragments of code reusable in different components.

* It has nevertheless a major difference: composition API hooks are called once and uses Vue's reactivity system while react hooks can be triggered multiple times during rendering

As an example, we could write this code fragment for google authentication logic:

// src/composables/useGoogleAuthentication.js

import { onMounted } from 'vue'

export default function useGoogleAuthentication() {
  const initGapiClient = async () => {
    /* initialization of google api client */
  }
  const loginWithGoogle = async () => {
    /* logic to open google popup, retrieve the authorization code and send it to backend */
  }

  onMounted(initGapiClient)

  return {
    loginWithGoogle
  }
}

Then, to retrieve the pieces of information required on our login page, the useGoogleAuthentication options can be merged into our page options (notably data, methods, and computed data but also lifecycle hooks). This merge is managed with the setup LoginPage component option.

// LoginPage.vue
<template>
  <div class="login-page">
    <button-login label="Login with Google" :icon="googleIcon" @click="loginWithGoogle"/>
  </div>
</template>

<script>
import ButtonLogin from './components/ButtonLogin.vue'
import useGoogleAuthentication from './composables/useGoogleAuthentication'


export default {
  name: 'App',
  components: {
    ButtonLogin
  },
  data () {
    return {
      googleIcon: { src: 'path/to/google-icon.png', alt: 'google icon' },
    }
  },
  setup() {
    const {loginWithGoogle} = useGoogleAuthentication();
    return {loginWithGoogle};
  },
}
</script>

* In order to prevent the loginWithGoogle method from triggering twice you will need to remove the custom click event in LoginButton.vue (@click="$emit('click') is no more necessary in Vue 3, the root element of a component will inherit the attribute)

As you can see, this solution prevents any implicit dependencies or naming collisions as you now need to clearly import the reusable part of your code. Reusing the authentication logic in different parts of the application is straight-forward.

This solution seems to be fitting our needs, right?

Unfortunately, we were working with a framework that was not yet supporting Vue 3. We had to find a workaround.

Good to know: As planned in the vue 3 roadmap, and especially because the support for IE may be stopped for Vue 3, some backport compatible features will be added to Vue 2.7. Among which the Composition API (the composition api plugin will be merged to Vue 2 Core). This change may enable Vue 2 users to benefit from the composition API features.

Solution: Use scoped slots

Looking a bit deeper in the Vue 2 documentation, we found the v-slot attribute (formerly named slot-scope). This attribute allows us to retrieve data or methods from a child component and pass it to the children of the component. Thus, for each connection method, we created a component including the connection generic logic and making the login method available to its children.

Here is an example with the GoogleAuth component. It encapsulates the google login logic and passes on loginWithGoogle method to its children.

// components/GoogleAuth.vue

<template>
  <div>
    <slot :loginWithGoogle="loginWithGoogle" />
  </div>
</template>

<script>
export default {
  data () {
    return {
      auth_code: ''
    }
  },
  mounted () {
    /* init gapi client */
  },
  methods: {
    loginWithGoogle () {
      /* logic to open google popup, retrieve the authorization code and send it to backend */
    },
    onFailure () {
      this.$emit('auth_error', { error: 'google error' });
    },
    onSuccess () {
      this.$emit('auth_success');
    },
  }
}
</script>

Then, the component GoogleAuth will wrap a button login and pass the login method to it. (Two other similar wrapping components for Facebook or email have to be created)

// LoginPage.vue

<template>
  <div class="login-page">
    <google-auth
      @auth_error="() => {/* manage authentication errors */}"
      @auth_success="() => {/* manage authentication success (e.g., navigate to another page) */}">
      <template v-slot="{loginWithGoogle}">
        <button-login label="Login with Google" :icon="googleIcon" @click="loginWithGoogle"/>
      </template>
    </google-auth>
  </div>
</template>

<script>
import ButtonLogin from './components/ButtonLogin.vue'
import GoogleAuth from './components/GoogleAuth.vue'
export default {
  name: 'App',
  components: {
    ButtonLogin,
    GoogleAuth
  },
  data () {
    return {
      googleIcon: { src: 'path/to/google-icon.png', alt: 'google icon' },
    }
  },
}
</script>

This solution answered our needs as it:

encapsulates the login logics in reusable components
splits the responsibility of the UX component (button) and the functional behavior (login logic)
doesn't create any uncontrolled side effects (each login method is scoped at the level of the button and not the page)

Conclusion

All the examples presented above allow us to compare different options to write and share component logic in Vue JS. In your project, you may choose a different option depending on the following criteria:

Criteria	JS Functions	Mixins	Scoped slots	Composition API
Encapsulate and reuse logic	✅	✅	✅	✅
Access to component options such as lifecycle hooks	❌	✅	✅	✅
No implicit or magic dependencies	✅	❌	✅	✅
No data collision	✅	❌	✅	✅
Readibility	✅	❌	✅ small component ❌ big component	✅
Where can shared logic be used ?	template & component	template & component	template	template & component
Available in Vue 2	✅	✅	✅	〜
Available in Vue 3	✅	✅	✅	✅

In a nutshell, for medium or big projects in Vue3, I would highly recommend using the composition API as it enables to write readable and reusable component logic anywhere in the code (including outside of components). For Vue 2, use scoped slots (for now) as it is a pretty straight-forward way to share component logic in Vue 2 without the drawbacks of mixins. It is even the best option for code reuse when dealing with a reusable component that provides a customizable UI in both Vue 2 and Vue 3.

A Newbie's Guide to React Query

Thu, 03 Jun 2021 00:00:00 GMT

Let's say you're building an application that relies heavily on displaying and interacting with database information (super rare situation as a dev, I know). You're also pretty good at React and Redux, so you start building your app with those libraries.

Next, you make an API request for a dataset and put it in your Redux store. Then you render the Redux store data.

Yep, I'm following. So far so good

Next comes manipulation of that data! You build out some actions that handle adding, editing, and deleting data.

Yeah, that's pretty standard!

Then, you trigger another GET of the data. React re-renders for you and you're now displaying the updated data set!

But what do you do if the API you're working with is slow? Getting that updated data back could lead to a delay in rendering the latest data, leading to a subpar user experience.

Hmm... guess I could update the store data?

True, but what then happens if the API call was not successful? Now the data in the Redux store is out of sync with the API data. How should we handle this situation? Keep track of the previous state to fall back to? How would we write up the logic for that?

Where is your god now?

Well, it seems we've stumbled upon a common problem — so common it feels like there should be a simple solution. Luckily for us, some great solutions are out there. Let's dive into my favorite one!

What is React Query and how does it solve our problem?

The React Query site claims the following: "Performant and powerful data synchronization for React. Fetch, cache and update data in your React and React Native applications all without touching any 'global state'."

Sounds pretty good, right? As we've discussed earlier, holding data in our global state can cause a host of issues. As React Query's author Tanner Linsley says, "It's time to break up with your global state!"

Let's take a look at some sample code:

Now this is where the magic happens: the first argument in useQuery, "todos", represents the query key.

Okay, but it just looks like a random string to me..

The query key is core to React Query — it will automatically cache query data based on these keys. Thus, you should always write and use query keys that are unique to the query's data. While this example's query key is a simple string, the query key can also be an array that contains strings, variables, and even objects!

Nice! Now we're good to go, right?

Almost! Now we're using React Query and have access to our data. And as we just learned, caching is already built-in! But are we already using it? Do we have to enable it somehow?

React Query Cache Basics

As it turns out, the code we saw above doesn't use React Query's cache just yet. First, we have to assign a value to the staleTime property.

...huh?

Think of staleTime as a (millisecond) countdown timer. When the timer expires, React Query then marks the query data it has cached as "stale." As a result, the query data is refetched in the background at the next available opportunity (see the docs for a list of situations that can trigger this). You can even set this value to Inifinity, for which it will never be stale!

Gotcha. So, why again did we need to set it in the first place?

If we peek at this page in the docs, we find that React Query is configured with "aggressive but sane defaults," one of which is the default setting of 0 for staleTime. That's pretty aggressive if you ask me!

Alright, now show me some code!

We've now assigned the value of one minute to staleTime! This way, the next time a query is fired with the query key 'todos' within the next minute, it will return the cached data. Woohoo!

Nice! But what if the data changes within that minute though?

That's right! Since our data fetching has this staleTime property, we may not see my new data for at most another minute — or even longer if we set a longer staleTime! So what do we do now?

As you've probably guessed, the clever folks who write and maintain React Query have got us covered. We have an invalidateQueries method on the QueryClient that we can use that invalidates query data by marking it stale. However, we can do even better!

React Query Mutations

Enter the almighty React Query mutation! Mutations give us a way to create, update, or delete data — and allow a lot of flexibility with how the query data is affected as a result.

Sounds great, where do I sign up?

We can get started using React Query mutations with the useMutation hook. The useMutation hook provides many useful options but we'll be focusing on a few: onMutate, onError, and onSettled.

onMutate is fired just before the actual data manipulation function — updateTodo, in the example below.

Wait, what's happening here? setQueryData?

Good eye! We've also made use of yet another handy React Query feature here: Optimistic Updates. Remember how we'd have a delay in rendering the latest data if our API response came back too slow?

Yeah...

We solved that here simply by calling the setQueryData method! The first argument is the query key, and the second argument is a function where we have access to the old query data. The function expects you to return new data, so we can return the old data with our newToDo appended to it.

Great, so we're done now!!

Not so fast! What we've done so far is improve the rendering of new data when things go well and we are on the happy path. But, what happens if our create, update, or delete calls don't go as planned?

Again, thanks to React Query, we have the ability to easily snapshot the old data and fall back to it! The first step is to "snapshot" it by saving it in a variable, then returning it, like so:

Next, we'll simply use that snapshot in our onError function and call setQueryData with it!

Great, now we're done! ...right?

Yep! Just kidding, we're not done yet! There's one more thing we should do: remember the invalidateQueries method we briefly mentioned before? This marks the data as stale and tells React Query that the query data tied to this particular query key should be refetched. The onSettled function gets fired at the end of a mutation whether the result was successful or not. This seems like a good place to mark the data as stale:

Here's what we have now. This will handle updating the todo item, roll back in case of an error, and mark the dataset as stale to trigger a refetch. Pretty neat if you ask me!

This piece of code does all that? Awesome!

Key takeaways

When implementing React Query, remember to always use the same query key for its matching data set
If you want to make use of caching, be sure to set a staleTime with your query
When adding new, editing, or deleting data, use a mutation to help you handle synchronization and fallback logic

If your application has a need for data synchronization, why not switch to using React Query?

Concluding thoughts

React Query gives us the luxury of not having to think too hard about how to keep our data in the front end up to date with our database information. Global state management is a great tool, but a lot of thought and careful planning has to go into handling edge cases and logic if things stray from the happy path. React Query makes handling all of this much easier, and even comes prepackaged with many powerful features like caching!

Looking to get a bit more in depth? Let me know if you’d be interested in a part 2 in the comments below!

How to Prevent Springboot Crashes after a Checkout

Sat, 15 May 2021 00:00:00 GMT

At Theodo, I'm working on a project where we need to start 5 APIs simultaneously where 40 developers are continuously pushing new lines of codes and our stack is relying on Spring Boot. Unfortunately one of our favorite action during our daily dev routine is just crazily complicated: 10% of the time, checking out from a branch to another one break the build of our APIs. It's a real pain point and we lose hours in debugging for something that is not a bug!

My problem is: How to checkout smoothly without worrying about API compilation being different between branches?

TL;DR

This script composes the post-checkout hook to automate how to cleanly restart your Spring Boot APIs:

#!/usr/bin/env bash
IFS=$'\n\t'

WORKDIR=$PWD
PREVIOUS_HEAD=$1
NEW_HEAD=$2

APIS="
users-api
payment-api
mailing-api
"
for nameApi in $APIS;
do
  cd ./$nameApi/src/

  GIT_DIFF=$(git diff $PREVIOUS_HEAD...$NEW_HEAD --relative)
  cd $WORKDIR
  if [[ ! -z "$GIT_DIFF" ]]; then
    if [[ "$(docker-compose ps $nameApi | grep Up)" ]]; then
      rm -rf ./${nameApi}/target
      docker-compose restart $nameApi
    else
      rm -rf ./${nameApi}/target
    fi
  fi
done

What happens when you checkout?

Let's take a simple example; I have to review the code of feature_branch_2, I thus have to jump often from feature_branch_1 to feature_branch_2.

The diagram above illustrate the differences between the 2 branches :

Model 1 doesn't exist in the second branch
Model 2 has some changes between the two branches

During the checkout, the change of branch can be brutal for Spring Boot and the automatic reloading does not manage to notice all the changes. Some of them may be missing from the compiled files and the server may crash.

My team was using a trick to circumvent the problem, it was performed through 3 painful operations :

Stop the server
Clean the builded target files
Rebuild the docker image and start the server (rebuilding the docker image included a target files build)

After some tests, we optimized the first and third step and only had to do the following :

Clean the builded target files
Restart the server

We reduced the number of steps but who wants to do 2 (if not 3) commands like this at each checkout?

How to automate this routine?

The best way I chose to automate was the post-checkout action that is native in Git versioning system. The idea is simple, after each checkout, a bash script is automatically executed based on the difference between the start and end branches of the checkout. This script is the following:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

WORKDIR=$PWD
PREVIOUS_HEAD=$1
NEW_HEAD=$2

APIS="
users-api
payment-api
mailing-api
"
for nameApi in $APIS;
do
  cd ./$nameApi/src/

  GIT_DIFF=$(git diff $PREVIOUS_HEAD...$NEW_HEAD --relative)
  cd $WORKDIR
  if [[ ! -z "$GIT_DIFF" ]]; then
    if [[ "$(docker-compose ps $nameApi | grep Up)" ]]; then
      rm -rf ./${nameApi}/target
      docker-compose restart $nameApi
    else
      rm -rf ./${nameApi}/target
    fi
  fi
done

The idea is to execute the clean and the restart for each API of my application, only if the sources' files from one branch to another contain differences.

The clean allows for a clean start of the API whether it is up or down. Additionally, I restart the API if it was already up before the checkout!

To make this script working on your project, you just need to copy/paste it in a file named .git/hooks/post-checkout.

What about the first lines?

I got into the habit of starting my bash scripts with the set -euo pipefail option allowing it to stop at the correct line if there is an error. It's easier to debug if required.

Also, the variable IFS=$'\n\t' allows the terminal to set the internal field separator.

To go further

The major problem with putting your script in the .git folder is that you can't make the most of Git to version and share it with your whole team. Indeed, this folder is often ignored in the repo and has no history. In fact someone else in the team brilliantly discovered the Husky library. With this library, you can put your script in a versioned folder (let's call it scripts at the root of your project) and configure Husky to call it whenever you want (let's say at the post-checkout event obviously). You can then have this type of configuration in your package.json:

{
  ...,
  "husky": {
    "hooks": {
      "post-checkout": "./scripts/post-checkout-hook.sh $HUSKY_GIT_PARAMS"
    }
  },
  ...
}

And if you want more with the Git hooks, there are plenty of other applications. For example, you can use the pre-commit hook to run the linter and format the code before sharing it with others! How many times had we changed a few things in a file and the saving started formatting everything because the previous developer who worked on the file didn't set up the IDE's linter...

You also can deal with other hooks proposed by Git, Reynald Mandel made a great example of post-merge hook use.

So what?

To talk about the limits, the script is executed each time we checkout and if the git diff resulting in a huge number of files, the execution can use a lot of memory. Just not that it's limited by the architecture of the script, we actually don't check each file, we just check if there is a difference or not.

Finally, with this hack, I reduced the frustration of the developers in the team, and we gained 40 hours of work per week at the scale of all the team (40 developers). This also represents the sobering saving of a developer on our client's bill!

Spice up Your Website with a DocuSign Electronic Signature Embedded Tool

Fri, 14 May 2021 00:00:00 GMT

Introduction

Integrating electronic signing in a website is not always so easy. If you ever get tired of spamming users who already have an overflowing mailbox with DocuSign emails, then you have landed on the right article. Let me show you how to replace them with an elegant embedded tool that will boost your platform.

This article addresses the following problems:

Finding a simple way to make people sign a disclaimer before accessing the content of a certain page;
Using DocuSign in a modal;
Embedding everything on your website.

Why this one-page architecture?

The goal of this architecture is to make sure users have to go through the disclaimer check before accessing the page. If you've got two different pages for that purpose, any decent attacker could bypass it:

Yup, it's almost as simple as that...

Why not using DocuSign emails?

When developing a feature using DocuSign, you will probably have to pick one of these two options:

Embed DocuSign signing;
Send DocuSign emails (which seems to induce less coding thus sounds more attractive).

After going through a lot of pain playing around and debugging unsent emails, I figured that not having to worry about unsent, unread, or spam-categorized emails was a fair point for the first option. The second reason is simple: it's easier to track issues, as everything - or almost - is embedded within your website. Last but not least, it came up that having a linear path would boost the UX by removing external dependencies.

Not paramount but worth mentioning, it also makes it easier to perform E2E tests on this feature.

In overall, this solution was:

Simple enough to be quickly implemented;
Customizable enough to fit all my different cases;

Installation

Simply run pip install docusign-esign to get started.

Let's break it down

We will assume in this article that the user has not signed the disclaimer yet.

Server-side

As shown in the schema above, the backend falls in two steps :

Defining the DocuSign envelope

The backend receives user information (email, name, ...) from the frontend. Then, we use the DocuSign Python Client to define our DocuSign envelope, i.e which document(s) will be part of the envelope, where they should be signed, who should sign them, and so on. This results in an envelope definition that we send to DocuSign to create it.

Getting the signing URL

DocuSign returns an envelope ID corresponding to the created envelope. It is used to start the signing ceremony, meaning that it is sent to DocuSign to generate a signing URL. Ultimately, this signing URL is sent back to the frontend.

Bear in mind that we can pick at this stage the redirection URL, i.e to which URL the user will be redirected once the signing ceremony is finished (necessary to understand our trick!).

Macro implementation (view)

Following this logic, the implemented view should create a DocuSign envelope and return its signing url to whomever calls the related endpoint.

docusign_service = DocuSignClient()

class DocuSignViewSet(APIView):
    def get(self):
        envelope_id = docusign_service.call_docusign_create_envelope(
            full_name=full_name,
            email=email,
            contact_crm_id=contact_id,
            document_path=document_path,
            document_name=document_name,
        )
        signing_url = docusign_service.start_signing_ceremony(
            envelope_id=envelope_id,
            full_name=full_name,
            email=email,
            contact_crm_id=contact_id,
            redirection_route=<CUSTOM_REDIRECTION_ROUTE>,
        )

        return Response(signing_url)

Micro implementation (DocuSignClient class)

Click here to discover details about its implementation

To make it easier, let's create a DocuSign class with all necessary methods. This will let us keep the same valid token during the whole process and renew it if needed.

import jwt
from docusign_esign import *

class DocuSignClient:
     token = None

Now, to communicate with the DocuSign API, what we first need is a way to get a valid EnvelopesApi object, i.e that has a valid token and correct parameters (pathname and headers), so let's add these two methods:

     def get_client(self):
         api_client = ApiClient()
         api_client.host = <MY_DOCUSIGN_API_BASE_PATH>
         api_client.set_default_header("Authorization", f"Bearer {self.token}")

         return api_client

     def get_envelope_api(self):
         api_client = self.get_client()
         envelope_api = EnvelopesApi(api_client)

         return envelope_api

However, because we want the token to be valid and to automatically renew if not, I added the following pre-check:

     def get_client(self):
         if not self.is_token_valid():
             self.token = self.renew_token()

        # ...

As you will see below, this check is essentially a call to the DocuSign API using the current token to verify whether it is valid. If it is expired or invalid (401 status code), we renew it. We use a single login for all users in the platform. Therefore, we use the JWT Grant authentication as suggested in the documentation.

     def is_token_valid(self, method="GET"):
         headers = {"Authorization": f"Bearer {self.token}"}
         response = requests.request(
             method, f"{<MY_DOCUSIGN_API_BASE_PATH>}/v2.1/accounts/", headers=headers
         )

         if response.status_code != 401:
             return True

         return False

     def renew_token(self):
         current_datetime = datetime.datetime.now()
         jwt_token = jwt.encode(
             {
                 "iss": <MY_DOCUSIGN_INTEGRATION_KEY>,
                 "sub": <MY_DOCUSIGN_API_USERNAME>,
                 "name": <NAME>,
                 "aud": "account-d.docusign.com",
                 "iat": current_datetime,
                 "exp": current_datetime + datetime.timedelta(minutes=30), # validity duration of 30 minutes
                 "scope": "signature impersonation",
             },
             key=<MY_DOCUSIGN_PRIVATE_KEY>.encode(),
             algorithm="RS256",
         )

         headers = {"Content-Type": "application/x-www-form-urlencoded"}
         url = f"{<MY_DOCUSIGN_BASE_URL>}/oauth/token"
         payload = {
             "grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
             "assertion": jwt_token,
         }
         response = requests.request("POST", url, headers=headers, data=payload)

        return response.json()["access_token"]

Believe me, all these methods will simplify your life. You'll only have to call get_envelope_api() to get a functional EnvelopeApi with a valid token.

Once done, we can then add methods to create a DocuSign envelope, which is roughly a container for documents that you send to a recipient for signing.

     def make_docusign_envelope(
         self,
         signer_full_name,
         signer_email,
         contact_id,
         document_path,
         document_name,
         anchor_y_offset="0",
         anchor_x_offset="0",
     ):
         with open(document_path, "rb") as file:
             content_bytes = file.read()
             base64_file_content = base64.b64encode(content_bytes).decode("ascii")

         document = Document(
             document_base64=base64_file_content,
             name=document_name,
             file_extension="pdf",
             document_id=1,
         )

         signer = Signer(
             email=signer_email,
             name=signer_full_name,
             recipient_id="1",
             routing_order="1",
             client_user_id=contact_id,
         )

         sign_here = SignHere(
             anchor_string="/sn1/", # the anchor string to indicate where to sign on the PDF
             anchor_units="pixels",
             anchor_y_offset=anchor_y_offset,
             anchor_x_offset=anchor_x_offset,
         )

         signer.tabs = Tabs(sign_here_tabs=[sign_here])

         envelope_definition = EnvelopeDefinition(
             email_subject="Please sign this document",
             documents=[document],
             recipients=Recipients(signers=[signer]),
             status="sent",
         )

         return envelope_definition

The make_docusign_envelope() method allows you to design a fully customizable envelope, by defining all components such as the document content, signers, locations where a signature has to be attached... You can then use it as a template like so:

     def call_docusign_create_envelope(
         self, full_name, email, contact_crm_id, document_path, document_name,
     ):
         envelope_api = self.get_envelope_api()
         envelope_definition = self.make_docusign_envelope(
             full_name, email, contact_crm_id, document_path, document_name
         )

         results = envelope_api.create_envelope(
             <MY_DOCUSIGN_API_ACCOUNT_ID>, envelope_definition=envelope_definition,
         )

         envelope_id = results.envelope_id

         return envelope_id

Indeed, call_docusign_create_envelope will make use of the previous method to create an envelope based on this template and returns an envelope ID that can be used to start the signing process (a.k.a ceremony).

    def build_docusign_redirection_url(self, redirection_route, options):
         return <MY_FRONTEND_URL> + redirection_route + options

     def start_signing_ceremony(
         self,
         envelope_id,
         full_name,
         email,
         contact_crm_id,
         redirection_route,
         authentication_method="none",
     ):
         envelope_api = self.get_envelope_api()
         return_url = self.build_docusign_redirection_url(redirection_route, envelope_id)
         recipient_view_request = RecipientViewRequest(
             authentication_method=authentication_method,
             client_user_id=contact_crm_id,
             recipient_id="1",
             return_url=return_url,
             user_name=full_name,
             email=email,
         )

         results = envelope_api.create_recipient_view(
             <MY_DOCUSIGN_API_ACCOUNT_ID>,
             envelope_id,
             recipient_view_request=recipient_view_request,
         )

         return results.url

Note that the custom redirection url built by build_docusign_redirection_url is the one that will be used frontend-wise once the signing ceremony is completed, i.e when the user clicks on the "Finish" button. Completing the ceremony will also update the envelope status in DocuSign to completed.

Client-side

Add DocuSign in an iframe

When the user accesses the restricted page, the backend is called to get a signing URL. Our modal contains an iframe that will use this URL to embed DocuSign content, meaning DocuSign envelope signing portal with all documents which have to be checked.

export const DocuSignModal: React.FC = () => {
  const { signingUrl } = getDocuSignSigningUrl(); // calls the previously created DocuSignViewSet

  return (
    <Modal>
      <iframe src={signingUrl} />
    </Modal>
  );
};

Use the redirection link to update the parent component

As seen previously, once the user signed all necessary documents and clicked on DocuSign "Finish" button, the URL provided when the signing ceremony started is used to redirect the user. The issue here is that the redirection will occur within the iframe since it is a different browsing context. And of course, we do not want that.

And now the trick!

Remember what we said about having free hands on the redirection URL? Well, we used this feature to build a URL such that it points to a page containing a shadow component and contains the corresponding envelope id. It will look like so: https://my-website.com/my-redirection-page/[ENVELOPE_ID]. Once rendered, this shadow component will post a message to the parent component (i.e the modal). This message contains the envelopeId of the created envelope, which can be found in the URL.

export const MyShadowComponent: React.FC = () => {
  const router = useRouter();

  useEffect(() => {
    window.top.postMessage(
      JSON.stringify({
        success: true,
        envelopeId: String(router.query.envelopeId),
      }),
      window.location.origin
    );
  });

  return null;
};

This message sent from the iframe will be listened to by the parent component (the page itself), which can then perform actions such as closing the modal, refreshing, routing to another page...

// event.origin and window.location.origin have to be compared for security reason
const isSameOrigin = (event: MessageEvent) =>
  event.origin === window.location.origin;

useEffect(() => {
  const handler = (event: MessageEvent) => {
    if (event.data && isSameOrigin(event)) {
      const { success, envelopeId } = JSON.parse(event.data);
      if (success) {
        // Do something such as closing the modal, saving the envelopeId, ...
      }
    }
  };
  window.addEventListener("message", handler);

  return () => window.removeEventListener("message", handler);
}, []);

You're done!

Conclusion

I hope you found interest in this article which highlights a simple way to embed DocuSign in your website. There are still many ways to improve and customize this flow, with additional features such as:

Saving in the backend users who have already signed so that they are not asked to sign again later ;
Tracking documents versions.

Hope that was useful, cheers!

Useful links

How to Check Your Website on Mobile in the Local Environment

Wed, 28 Apr 2021 00:00:00 GMT

At Theodo, we develop solutions for both mobile and desktop. Most of the time we use the chrome device toolbar to see the visual rendering of the website when we develop in the local environment. Then when the code is deployed, we also check in the development/staging environment that everything works correctly.

However, sometimes, this is not enough. For example, this is not applicable to debug a behavior that occurs only on mobile devices. The obvious solution: finding a way to check the website on mobile, in the local environment. There are several ways to do that and each has its perks and drawbacks. In this article, I will present you with those solutions to help you find the one that suits you best.

For you to find the best solution in your use case, I will evaluate each solution with the following criteria:

📱 ‍Provides an experience similar to the end-user interaction on mobile
📋 Easy access to logs
🔧 Easy to set up

Chrome device toolbar

Before you start, make sure that the feature you want to check cannot be checked with the Chrome device toolbar.

Chrome device toolbar is accessible by a single click on the mobile icon at the top of the Chrome inspector

It does what you expect and maybe more…

Chrome device toolbar is an obvious solution, known by most developers, that allows you to test responsiveness on a large variety of devices. But responsiveness is not the only thing you can test with the chrome device toolbar. All the available options are presented in great detail in the chrome documentation. If you need an incentive to go see for yourself, be aware that the device toolbar allows you to override geolocation, throttle your CPU and/or internet speed, display media queries… A less known feature of the Chrome device toolbar is that it properly sets the user-agent according to the device you choose. So if you implement a different behavior based on the userAgent, you will be able to see it.

But Chrome device toolbar is not an emulator

However, as complete as it is, the chrome device toolbar is still limited. Here is a quick list of things you may need, and that is still unavailable with this tool:

The fluidity of scrolling and swiping. One of the main differences between desktop and mobile is the way the user interacts with the device. And this is a big drawback of the Chrome device toolbar. Your scroll could be lagging like hell on a mobile device, you would not be able to see it.
Opening/closing of keyboard and focus. Don't forget that the Chrome device toolbar remains more or less a desktop website visualizer that can take the shape of a mobile device. It won't display the behaviors that are particular to mobile such as the opening of a keyboard.
Running code that is specific to an operating system. A video player designed to run on IOS won't run in the chrome toolbar even if you selected an iPhone.

Solution evaluation

📱 Close to end-user experience: Mediocre

❌ Chrome device toolbar is not an emulator.

📋 Access to logs: Excellent

✅ No need to change development flow to retrieve logs. Everything you need is directly available in the browser.

🔧 Easy to set up: Excellent

✅ A click on a button in the inspector.

Chrome device toolbar is enough for basic responsiveness but what if we need to access our website / debug on a real mobile device ?

Using your local network to access your website in the local environment

There is a quick solution that doesn't require any third-party software and that allows you to connect to your website from your mobile on your local network. All you need is a mobile device and a computer connected to the same network, i.e. a wifi network, or a shared mobile internet connection.

The principles

As you probably already know, when setting up the services for your application, you give them an address which on your local environment is localhost + port number. You can make your website accessible to all the users on the local network by exposing it to your address on this network.

The reason for this is rooted in the difference between localhost and your local IP address. Localhost is an internal address in your computer used by the loopback interface, a virtualized network interface. Localhost is only used inside your computer. On the other end, the local IP address is an address given by your router for your computer to communicate with other devices on the local network.

How to set it up

Identify your local Ip address. On Windows, you can open the terminal and execute ipconfig. On Mac and Linux, the command to execute is ifconfig. But maybe you won't even need to do that. For example, when starting a React App, the local IP address is displayed in the console.

You can see your local IP address when launching a React App as "On Your Network"

If you only have a frontend, you are already good to go. Just enter the local IP address in the URL bar of your browser with the right port, and you will see your website.
If you have a backend, you should replace your backend adress(es) called by the frontend from

http://localhost:<your_backend_port>

http://<your_local_ip_address>:<your_backend_port>

Your website is now accessible at the following address on both your desktop device and to any device connected to the local network:

http://<your_local_ip_address>:<your_front_port>

A nice addition to this is to add a script such as the one below in your package.json to automatically fetch your local IP address and update your frontend/backend URLs in your environment files:

#grepping the local ip address
LOCAL_IP=$(ifconfig
| grep -Eo 'inet (addr:)?([0-9]*\\.){3}[0-9]*'
| grep -Eo '([0-9]*\\.){3}[0-9]*'
| grep -v '127.0.0.1');
# using a .env.mobile file as a template
# and moving the output into .env file
sed \"s,frontend_url,$LOCAL_IP,g\" .env.mobile > .env;
# starting the react app with the updated file
react_app start

Make CORS allowed host changes if needed. You will only have to do the same replacement as the previous point for your front-end address.

How to see the logs

Now that you are accessing the website in your local environment on a mobile device, the next step would be to see the logs of your device so you can develop and debug your application correctly.

MacOS ⇔ IOS. For this, you can use the Safari development tools. Connect your device to your computer using a USB cable and when asked if you trust the computer, answer yes. Open Safari and click on the 'Develop' tab

You can access your mobile logs by opening the Safari Browser. Select first the 'Develop' tab then the 'name of your device' and finally the 'name of the page'
Mac/Windows/Linux ⇔ Android: Google is providing a tool to access logs by connecting your mobile to your computer using a USB cable .
Windows/Linux ⇔ IOS: It seems there is no obvious solution for this currently.

A versatile solution that covers (almost) any use case

What is great about this solution is that it allows you to test directly on your mobile device and also that it works not only for a frontend but also for a "complete website" with a/multiple backend and a database as well.

But:

This solution requires updating all the addresses throughout your application. However, this can be easily solved by adding a script to your package.json
Some third-party API requires HTTPS and won't work with this setup.
Your router may change your local IP address from time to time.

Solution evaluation

📱 Close to end-user experience: Good

✅ You are debugging directly on your mobile device.
❌ You can only use this method on the devices you own, so you may overlook some edge cases on other devices.
❌ You also may encounter errors if HTTPS is mandatory for some APIs you use. In that case, see my next solution.

📋 Access to log: Very good - No Logs

✅ The same quality of logs as when you are developing for computer users...
❌ ...Except for the IOS + Windows/Linux duo.

🔧 Easy to set up: Very good

✅ Relatively easy to put in place and to automatize
✅ You don't need to install anything.
❌ A bit time-consuming to set up the first time

Now we can access our website on a real mobile device. But what if we need HTTPS ?

Using LocalTunnel to access your website in the local environment from your mobile with HTTPS

What is LocalTunnel?

LocalTunnel is a service that allows you to expose an endpoint on both a public HTTP and a public HTTPS address.

If you don't know localTunnel, maybe you are more familiar with ngrok its best-known counterpart. However, the main advantage of localTunnel over ngrok is that you don't have a limit on the number of endpoints you can create whereas you have to pay as soon as the second one for ngrok.

How to set it up?

You have two options to use LocalTunnel on your project

Start the tunnels using the command line using: lt --port <port>

Running localTunnel using the command line is pretty straightforward
Using the npm package directly in a node project

const localtunnel = require("localtunnel");
const myTunnel = await localtunnel({ port: 3000 });
console.log(myTunnel.url);

You will still have to update your URLs in your configuration file (same principles as for the local IP address). I personally prefer the second solution because the address returned by LocalTunnel is random and the task of updating the URL for your backend/frontend in your configuration files can quickly become annoying. I developed a quick example of an independent node project that sets up your configuration files when launching the app.

Concerning logs, they are still available using the same principles as for the local IP address.

The ultimate solution to local development on mobile?

Pros:

You are making your website accessible to anyone on the internet and not only on your local network anymore. You want your client/another developer/your manager to see the new feature before deploying it, this is now possible!
LocalTunnel provides an HTTPS address so you won't have any issue related to HTTP anymore.

Cons

Complex to set up. Each time you launch localTunnel, the addresses provided are different which makes it a bit tricky to set up with a script even though it remains possible.
You can only test on the devices/browser versions you have access to.
Even if there is little chance of anything happening you are still making your website public. Make sure you don't display any critical information.

Solution evaluation

📱 Close to end-user experience: Very Good

✅ Directly on your mobile device
✅ In HTTPS
❌ Only the mobile you own / the browser version you have access to

📋 Access to log: Very good - No Logs.

✅ The same quality of logs as when you are developing for computer users...
❌ ...Except for the IOS + Windows/Linux duo.

🔧 Easy to set up: So-so.

✅ Can be automated
❌ More time-consuming than the local-network solution

Now you can access to your website on a mobile device in HTTPS. But you may wonder what to do when you don't have the device / the browser version you want to check

Emulators/simulators

You may be tempted to use an emulator / a simulator to access your local website. Indeed, using Android Studios or Xcode Ios Simulator to access your website during local development is possible but here are the two main reasons why this is probably a bad idea:

Emulators are heavy. Your computer will soon be over-encumbered with mobile images
Emulators are slow. You have to imitate the behavior of smartphone components on your computer and it requires a lot of calculation power making your application slow/laggy in the emulator.

If you want to test a specific mobile or a specific browser version, your best bet is probably to use BrowserStack.

However, if you don't have any other options, here is a quick tutorial on how to use Android Studio and Xcode to emulate a smartphone and access your website in the local environment

Android

Install Android Studio from their developer website
Create an empty project
Add the device you want using the AVD manager in the "Tool" tab

You can choose a device from a large selection using the AVD manager by clicking on the "Create Virtual Device" button
In the new project launch the emulator on the device you want.
Once the emulator is running, open the google chrome application. You can access the localhost of your computer by entering the following address in the browser url bar.

10.0.2.2:<frontend-port>

If you have a backend, don't forget to change its adress in your front configuration to

10.0.2.2:<backend-port>

To have access to logs, go to chrome://inspect/#devices in google chrome browser while the emulator is running. On this page, you will see a list of devices connected to your computer, including the emulators. By clicking on "inspect", you will open a devTools window similar to the one you use for web development.

On the Chrome inspect page, you can see a list of device and for each device, a list of pages opened. Click on inspect for any page to open Chrome Dev Tools for this page

Ios

Only if you develop on macOS, you can use the simulator that is included in the Xcode development software:

Download Xcode from the Apple developer website
Create an empty project and click on the play button to launch the simulator. Once the simulated phone started, open the Safari browser application and go to localhost:<your_port>
To access logs, open Safari on your computer, and in the "Develop" tab, you will see the simulator and the opened page. Click on the page you want and a new window will open with all the inspection options for this page.

Click on the Safari "Develop" Tool and then on your simulated device to see the current open pages. If you click on a page, it will open a web inspection window

Solution evaluation

📱 Close to end-user experience: So-so

✅ Native behaviors can be tested for a large variety of devices
❌ Unable to check performance issue as the emulator itself is slow

📋 Access to log: Good - No Logs.

✅ The same quality of logs as when you are developing for computer users...
❌ Only for Chrome/Android or Ios/Safari
❌ Debugging with logs can be long because the pages takes a very long time to load / refresh

🔧 Easy to set up: So-so.

✅ Easy to use once Xcode / Android Studio are installed
❌ Very heavy and memory-consuming

Our decision tree is complete!

Takeaway

Two last things that can help you find the best solution for your project :

Don't go for the most technical solution if all you need is to test basic responsiveness. The chrome device tool is already very complete.
If you want to test complex behavior, you will probably know it beforehand. So set up your project for mobile testing at the beginning. You will lose far less time on the way.

How to develop on Shopify in 2021

Wed, 28 Apr 2021 00:00:00 GMT

As a software developer, I often hear that No-Code is the future of app development. That in a few years, automatically generated code will replace all the frameworks that we currently have. With my young experience in this field I'm quite sceptical, because I still believe that to truly answer a problem or a need you will always need something that can embrace it completely.

However, I was quite amazed by my last and first experience using one of the major tools in the No-Code business: Shopify.

Shopify became in the last few years the reference for online stores, and is now powering more than 18% of all eCommerce stores (according to BuiltWith). Their success can mostly be explained by the convenience of their product: by only using the platform, you can build a highly customized store, ready to use, in only 2 hours.

However, if you have very specific needs regarding your online store, you will quickly need a developer to be able to deep dive into Shopify's technical documentation. Therefore, I decided to write this article to summarize the different things that I learnt using Shopify as a developer, and how I used them in my project.

Here are the different topics that will be covered:

Front framework: Liquid
Embed a React App inside Liquid's engine
Shopify's API - Usage and Limitation
Development Workflow: Automatize your deployment with CircleCI, Github and ThemeKit

Front framework: Liquid

Even though Shopify has the specificity to be used as a headless ecommerce solution (using their API), the native behavior of Shopify is to use their embedded front system (Wordpress-like).

And for our project, we decided to stick with the classic front system as it allowed us to have an instant-ready website, giving the possibility to our client to test different configurations.

So, we had to deal with their own programming language: Liquid.

Liquid is an open-source template language written in Ruby. As many template languages (like Jinja (Python) or Twig (PHP)), it makes a bridge between data and HTML, in which you can add logic (if, while loops...). In Shopify's usage of Liquid, the data is the different entities related to your store (products, cart, collections...).

Before launching yourself into any Shopify development, I highly recommend you to read this introduction to the Liquid language, written by Keir Whitaker, a former Shopify Partner. It describes in an exhaustive way how Liquid is built and how Shopify use it for their front framework.

It also mentions one of the main downsides that we encountered during our project: Liquid doesn't have any concept of state, which can really be a pain point if you are willing to make your website a little bit interactive, by creating a real user experience. In our case, we wanted to create an interactive quiz, in which our users could answer a few questions about themselves, and push to them the best matching products.

In order to develop our feature inside Shopify's front engine, we decided to use React's framework, embedded inside a liquid template.

Embed a React App inside Liquid's engine

Our challenge here was to be able to use the ease of front development with React without creating a separated hosted webapp. Our idea was to develop the React App in the same repository, and then build the static files (as for production) and add them to the assets of a Shopify theme.

Here was our deployment script:

#!/bin/bash

APP_BUILD_FILE=build/index.html
THEME_FILE=theme_live/sections/quiz.liquid

echo 'Adapting html file for shopify asset urls'
sed -E 's/(<script src=")\/static\/js\/([a-zA-Z0-9._-]*)("><\/script>)/\1{{'"'"'\2'"'"' | asset_url }}\3/g' $APP_BUILD_FILE > $THEME_FILE

echo 'Replacing built javascript files in theme'
rm -f theme_live/assets/quiz*
cp build/static/js/* theme_live/assets/

Once a React App is built, the entry file is the index.html that contains all the script tags pointing to the js files of your app, divided in different chunks. The first command of our script catches all the script tags contained in the index.html file and then replaces them by a liquid syntax.

From:

<div id="root"></div>
<script src="/static/js/quiz_runtime-main.8923d629.js"></script>
<script src="/static/js/quiz_2.20b50bf0.chunk.js"></script>
<script src="/static/js/quiz_main.f9693573.chunk.js"></script>

We now have:

<div id="root"></div>
<script src="{{'quiz_runtime-main.8923d629.js' | asset_url }}"></script>
<script src="{{'quiz_2.b689c532.chunk.js' | asset_url }}"></script>
<script src="{{'quiz_main.d6725a08.chunk.js' | asset_url }}"></script>

The second command then removes the previous react JS files and replaces them with the new assets, in our quiz section.

However, this creates a separate app that can't interact with the liquid framework. Therefore, we needed to analyze how we could retrieve data from our Shopify store and display it to the user. Hopefully, Shopify has a great API, which allows you to interact with a lot of different elements of your stores.

Shopify's API - Usage and Limitation

This is where Shopify becomes very interesting. As said before, using the built-in Liquid framework allows you to create a ready-to-use store in only 2 hours. However, you'll sometimes find yourself in need to interact with your store data outside of your storefront, or maybe you want to only use Shopify as a backend, and make a completely custom frontend ,using gatsby or NextJS.

This gives you the possibility to move from a Liquid frontend to a much more customizable interface, by using Shopify's back API.

Shopify has 2 APIs available:

The Admin API (GraphQL and REST)
The Storefront API (GraphQL only)

Shopify's Admin API

The Admin API is the main API that allows you to interact with ALL the elements of your store. You can use it to follow your orders, products, payments... Well, pretty much everything. This API is mainly used by shopify's app developers, to extend the behavior of the basic Shopify back office.

In my case, I used it to create our product recommendation algorithm. As said before, our challenge was to create a quiz in order to advise our customers on a very specific market. We wanted to know:

The characteristics of the products they were looking for
Their family composition
What type of products they were looking for

Once the answers were given, our app then called the Admin API, in order to retrieve the best matching products. However, this is where I discovered the API's rate limitations. Indeed, our main goal was to consider all the selected product categories by our user (that could go up to 15 categories - the aim of our quiz was to create big buckets of products), and to propose in each category the best product.

We went for the GraphQL API because it allowed us to:

Make a single call with different product categories
Get only the data that we needed

However, GraphQL's API usage is limited by the cost of your query. This means that you cannot make a huge query in order to retrieve all your data at once. With classic Shopify, your query can have a maximum cost of 1000.

As we didn't have a lot of products per category, we limited the number of products per category to retrieve per 20, and limited to the maximum the data that we wanted to retrieve:

productType1: products(first: 20, query: "product_type:'productType1'") {
    edges {
        node {
            id
            productType
            tags
            priceRangeV2 {
                minVariantPrice {
                    amount
                }
            }
        }
    }
}
productType2: products(first: 20, query: "product_type:'productType2'") {
    edges {
        node {
            id
            productType
            tags
            priceRangeV2 {
                minVariantPrice {
                    amount
                }
            }
        }
    }
}

Once we retrieved the best matching products, we then sent back only the selected productIds, to give the possibility to our front to call the storefront API.

Shopify's Storefront API

Storefront's API is a way to expose your store's data to external apps (public or private), unauthenticated. The perfect usage of this API is when you want to use Shopify only as a merchant platform, and create your custom front platform (Shopify as a headless ecommerce solution).

It is important to remember that it's an unauthenticated API, meaning that once the Storefront's API is enabled, everybody can access the data that you exposed. However, you can control the data that you expose (you can only display the product details for example).

We used the storefront API for our store directly in our React App, in order to display the different products, and call the checkout to add the different items to the user's cart. We got inspired from the different adaptation of the buy-sdk adapters that you can find on Shopify's Github.

Development Workflow: Automatize your deployment with CircleCI, Github and ThemeKit

Last but not least, I will present the development workflow that we set up on our project.

Shopify has developed a command-line tool called Theme Kit, enabling you to interact with your shopify store and deploy code instantly.

Once Theme Kit is installed on your computer/server, you need to configure it inside your app folder, following this tutorial.

Theme Kit allows you to (in particular):

Preview your code as you code it on your shopify store theme using the theme watch command
Deploy code to one of your shop theme using theme deploy --env=development or any other name that you would have defined in your config.yml file.

development:
  password: ${SHOPIFY_PASSWORD}
  theme_id: ${SHOPIFY_DEV_THEME_ID}
  store: store-name.myshopify.com
  timeout: 1m0s
production:
  password: ${SHOPIFY_PASSWORD}
  theme_id: ${SHOPIFY_THEME_ID}
  store: store-name.myshopify.com
  timeout: 1m0s

Each developer had their own development theme (you can have up to 20 themes on one shopify store), and their was one "live theme" which was the production theme. The different useful variables were stored in a variables file, that Shopify's engine can interpolate. For further information, you can check Theme Kit configuration documentation.

Here was our developer workflow:

Get the latest code from the master branch using git
Push my code to my Shopify development theme using theme deploy --env=development
Make modification while using theme watch (so that you could see directly the changes on your Shopify theme)
Push your code, review it on github
Merge on master.
Once the code was merged, we had a CI/CD script that:
- Built the React App and moved the built files into the assets of our shopify theme
- Deploy the code to Shopify using Theme Kit

The last step was done using the following script in your CI:

echo 'Deploying runinng cd theme_live && theme deploy --allow-live'
cd theme_live && theme deploy --allow-live --env=production

The Shopify production Theme ID and user credentials were injected inside our CI configuration, to be able to deploy to the production environnement.

Conclusion

As you can see, on top of being a very easy-to-use product for non-tech people, Shopify is also a real state-of-art product. They thought of a simple workflow to automate your deployment, you can easily access your store using a highly documented API.

They created a real game changer when it comes to build an ecommerce website: you can start with a prototype using the classic Liquid templating product, then if you want to fully customize your storefront, you can change to a headless CMS version of Shopify using their API, and then you can even scale up by using their premium version Shopify Plus, allowing to manage different stores in different country.

And one thing that is important to notice: once your store is created, it's already usable. Shopify has their own secured payment platform, PIM management, and many other features that are required in every ecommerce store, which you don't need to develop anymore. And icing on the cake, by using Liquid's framework, your SEO will be optimized and fully manageable from your store.

I highly recommend each developer to take the time (with Shopify's 14 days trial) to build their own Shopify store, as it's a great skill to have in a world where the shopping experience is becoming more and more digitized.

How To Make Tree Shakeable Libraries

Tue, 13 Apr 2021 00:00:00 GMT

At Theodo, our aim is to build reliable and fast applications for our customers. Some of our projects include improving the performance of already existing applications. During one of these missions, we managed to reduce the bundle size of all our pages by a whopping 500KB Gzipped by tree shaking our internal libraries.

While doing so, we realized tree shaking is not something we can simply turn on and off. There are many factors that impact the tree shaking quality of libraries.

This article's aim is to provide an exhaustive guide on making an optimized tree shakeable library. If I had to summarize the procedure:

Check whether the library is tree shakeable by testing it against a known application in a controlled environment
Use ES6 modules so bundlers can detect unused export statements
Use the side effects optimization and make your library side effects free
Split the library logic in multiple small modules and preserve the library's module tree
Do not lose the module tree or the ES modules characteristics when transpiling your library
Use the latest version of a tree shaking capable bundler

What is tree shaking and why is it important?

From the MDN docs:

Tree shaking is a term commonly used within a JavaScript context to describe the removal of dead code.

It relies on the import and export statements in ES2015 to detect if code modules are exported and imported for use between JavaScript files.

Tree shaking is a way to achieve dead code elimination by detecting which exports are unused in our application. It is performed by application bundlers such as Webpack or Rollup but was initially implemented by Rollup.

So why call it tree shaking? One can visualize the application's exports and imports in the form of a tree. The healthy leaves and branches represent used imports of the application while dead leaves symbolize unused code separated from the rest of the tree. Shaking the tree would eliminate all our unused code.

Why is this important? Tree shaking can have a huge impact on your browser applications. The more code gets bundled in the app, the more time the browser will spend downloading, decompressing, parsing, and executing it. Removing unused code is thus of the utmost importance to make the fastest applications possible.

There are many articles and resources out there explaining tree shaking and dead code elimination. Here we will be focusing on libraries that are consumed by applications. A library is considered to be tree shakeable if a consumer application can successfully eliminate the unused parts of the library.

But before we try to make a library tree shakeable, let's first see how we can distinguish one.

Realize a library is not tree shakeable in a controlled environment

This may sound obvious at first glance. But I noticed a lot of developers assume their library is tree shakeable because it uses ES6 modules (more on that later) or because they have a tree shaking friendly configuration. Unfortunately, this does not automatically imply that your library is in fact tree shakeable!

This brings us to the relevant question: how can one efficiently check that a library is tree shakeable?

To do so, we need to understand two things:

It is the app's bundler that will ultimately tree shake the library's code, not the library's (if it even has a bundler at all!). After all, only the app knows which parts of our library are used.
The library's job is to make sure it can be tree shaken by the final bundler

To check whether our library is tree shakeable we will be testing it against a reference application in a controlled environment:

Create a simple application (reference app) with a bundler you know how to configure and that supports tree shaking (eg Webpack or Rollup)
Set the library you want to test as a dependency of the created application
Import only one element of the library and check the output of the application's bundler
Check that the bundled output only contains the imported element and its dependencies

This strategy will make the test independent of our existing applications. It makes it easier but also allows us to play with the library without breaking anything. It also makes sure the issue does not come from the application bundler configuration.

We will be doing this for our library called user-library that we will test against a user-app application bundled with Webpack. You may use whatever bundler you feel more comfortable with.

The user-library's code looks like this:

export const getUserName = () => "John Doe";

export const getUserPhoneNumber = () => "***********";

It just exports 2 functions in an index file that we can use via an NPM package.

Let's make our simple user-app:

package.json

{
  "name": "user-app",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "build": "webpack"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "webpack": "^5.18.0",
    "webpack-cli": "^4.3.1"
  },
  "dependencies": {
    "user-library": "1.0.0"
  }
}

Notice we are using user-library as a dependency.

webpack.config.js

const path = require("path");

module.exports = {
  entry: "./src/index.js",
  output: {
    filename: "main.js",
    path: path.resolve(__dirname, "dist"),
  },
  mode: "development",
  optimization: {
    usedExports: true,
    innerGraph: true,
    sideEffects: true,
  },
  devtool: false,
};

To understand the Webpack configuration of our reference application, we need to understand how Webpack does tree shaking. Tree shaking is performed using the following steps:

Identify the application entry file (determined in the Webpack configuration)
Create an application module tree by looping through all the dependencies and sub dependencies imported by the entry file
Identify for each module in the tree which export statements are not imported by the other modules
Eliminate the unused exports and their related code using minification tools like UglifyJS or Terser

This process in done only in production mode.

The issue with production mode is minification. It becomes very hard to see whether our tree shaking worked because we cannot actually see our named functions in our bundled code.

To get around this, we run Webpack in development mode while still identifying which code is unused and would be removed in production with the optimization property set to:

  optimization: {
    usedExports: true,
    sideEffects: true,
    innerGraph: true,
  }

The usedExports property allows Webpack to identify which module exports are not used by other modules. The other two will be discussed later in the article. For now, let's just say they improve the tree shaking quality of our application.

Our user-app entry file: src/index.js

import { getUserName } from "user-library";

console.log(getUserName());

Once bundled, we now analyze the output:

/***/ "./node_modules/user-library/dist/index.js":
/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((__unused_webpack_module, exports) => {

var __webpack_unused_export__;

__webpack_unused_export__ = ({ value: true });

const getUserName = () => 'John Doe';

const getUserPhoneNumber = () => '***********';

exports.getUserName = getUserName;
__webpack_unused_export__ = getUserPhoneNumber;
/***/ })

Webpack regroups all our code in a single file. Looking at the getUserPhoneNumber export, we notice that Webpack has marked it as unused. It will be removed in production mode while getUserName is exported as it is used by our index.js entry file.

The library is tree shaken! You may repeat this step but for multiple imports and look at the output code. The objective is to make sure unused code in the library is marked as unused by Webpack.

Everything looks fine for our very simple user-library. Let's make it a bit more complicated and as we do so, we will look at some tree shaking requirements and optimizations.

Use ES6 modules so bundlers can detect unused exports

Ok, this requirement is classic and well documented but is in my opinion a bit misleading. I oftentimes hear developers say we need to use ES6 modules so that our library can be tree shaken. While this statement is completely true, there is this misconception that using ES6 modules is enough to just make tree shaking work. Well, were it that simple, you definitely would not have gotten this far in the article!

However, using ES6 modules still is a requirement for tree shaking.

There are a lot of formats in which JavaScript can be bundled: ESM, CJS, UMD, IIFE and so on.

To make things simple, we will consider only two: Ecma Script Modules (ESM or ES6 modules) and CommonJS modules (CJS) as they are the most widely used for application libraries. Most libraries will use CJS modules because it allows them to be run in a node application (though Node now supports ESM). ES modules appeared way later than CJS in 2015 with ECMAScript 2015 (also known as ES6) and is considered to be the standardized module system for JavaScript.

CJS syntax example

const { userAccount } = require("./userAccount");

const getUserAccount = () => {
  return userAccount;
};

module.exports = { getUserAccount };

ESM syntax example

import { userAccount } from "./userAccount";

export const getUserAccount = () => {
  return userAccount;
};

The big difference between both is that ESM imports are static whereas CJS imports are dynamic meaning we could do the following with CJS but not with ESM:

if (someCondition) {
  const { userAccount } = require("./userAccount");
}

While this seems to be more flexible, it also means bundlers cannot make a valid application tree at compile or bundle time. The someCondition variable will be known only at runtime forcing the bundler to import userAccount in any case at compile time. This leads bundlers to just include all the CJS style imports directly in the bundle without being able to check whether the imports are actually used.

Let's modify our user-library to show this. And to make it a little bit more realistic, it will now have two files:

src/userAccount.js

const userAccount = {
  name: "user account",
};

module.exports = { userAccount };

src/index.js

const { userAccount } = require("./userAccount");

const getUserName = () => "John Doe";

const getUserPhoneNumber = () => "***********";

const getUserAccount = () => userAccount;

module.exports = {
  getUserName,
  getUserPhoneNumber,
  getUserAccount,
};

We keep the same entry file in our user-app so we do not use the getUserAccount function nor its dependency.

/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((module, __unused_webpack_exports, __webpack_require__) => {

const { userAccount } = __webpack_require__(/*! ./userAccount */ "./node_modules/user-library/dist/userAccount.js")

const getUserName = () => 'John Doe'

const getUserPhoneNumber = () => '***********'

const getUserAccount = () => userAccount

module.exports = {
	getUserName,
	getUserPhoneNumber,
	getUserAccount
}
/***/ }),

/***/ "./node_modules/user-library/dist/userAccount.js":
/*!*******************************************************!*\
  !*** ./node_modules/user-library/dist/userAccount.js ***!
  \*******************************************************/
/***/ ((module) => {

const userAccount = {
	name: 'user account'
}

module.exports = { userAccount }
/***/ })

All three exports still appear and are not marked by Webpack as unused. This is also the case for our userAccount file which will be included in the bundle.

Now let's look at the same example but with ESM by just replacing the require and exports syntax with their ESM counterparts.

/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {
/* harmony export */ __webpack_require__.d(__webpack_exports__, {
/* harmony export */   "getUserName": () => /* binding */ getUserName
/* harmony export */ });
/* unused harmony exports getUserAccount, getUserPhoneNumber */
/* harmony import */ var _userAccount_js__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(/*! ./userAccount.js */ "./node_modules/user-library/dist/userAccount.js");

const getUserName = () => 'John Doe';

const getUserPhoneNumber = () => '***********';

const getUserAccount = () => userAccount;

/***/ }),
/***/ "./node_modules/user-library/dist/userAccount.js":
/*!*******************************************************!*\
  !*** ./node_modules/user-library/dist/userAccount.js ***!
  \*******************************************************/
/***/ ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {

/* unused harmony export userAccount */
const userAccount = {
	name: 'user account'
};
/***/ })

Notice that getUserAccount and getUserPhoneNumber are marked as unused. But so does the userAccount export in the other file. Thanks to the innerGraph optimization, Webpack is able to link the userAccount import in the index file to the getUserAccount export. This allows Webpack to work recursively from the entry file and go through all of its dependencies to know which exports are unused in every module. Since Webpack knows that getUserAccount is unused, it can go and check its dependencies in the userAccount file and do the same work there etc.

ES modules allow us to look for exported code that is used or unused in our application explaining why this module system is so important for tree shaking. It also explains why one should use dependencies that export a ES module compatible build such as lodash-es, the ESM equivalent of the popular lodash library.

This being said, using only ES modules for tree shaking is still a suboptimal approach. In our example, we noticed Webpack worked recursively in each file to see whether exported code was used or unused. In this case, Webpack could have just totally ignored the userAccount file because the only import coming from that file is unused! This leads us to the notion of side effects discussed in the next part of the article.

To sum up this first part:

ESM is a requirement for tree shaking but is not enough to make it optimal
Make sure to always provide an ESM build of your library! If your consumers require both ESM and CJS builds, provide them both through the package.json with the main and module fields.
Make sure to use ESM dependencies when possible as they won't be tree shaked otherwise.

Use the side effects optimization to make your library side effects free

According to the Webpack docs, tree shaking can be split in two optimizations:

usedExports: Determine which exports of a module are used and unused
sideEffects: skip over modules that do not have any used exports and that are side effects free

To illustrate side effects, let's take the example that we used before:

import { userAccount } from "./userAccount";

function getUserAccount() {
  return userAccount;
}

If getUserAccount is unused, can the bundler assume that the userAccount module can be removed as well? The answer is no! userAccount could do all kinds of stuff affecting other parts of the app. It could inject some variables inside a globally accessible value, like the DOM for instance. It could also be a css module injecting style inside the document. But I think the best example is polyfills as we usually import them like:

import "myPolyfill";

Now this module definitely has side effects as it affects the code of the whole app as soon as it is imported. Bundlers will see this module as a possible candidate for removal as we are not using any of its exports. But removing this would break our app.

Bundlers like Webpack or Rollup will therefore see every module of our library as filled with side effects by default.

In our case, we know that our library is side effect free! We can therefore help our bundler by informing it of this. To do so, most bundlers can read the sideEffects property inside the package.json file. It is by default set to true when unspecified (there are side effects in every module of the package). You can set it to false (no side effects anywhere) or you may specify an array of files that have side effects.

We add this option to the package.json file of our library:

{
  "name": "user-library",
  "version": "1.0.0",
  "description": "",
  "sideEffects": false,
  "main": "dist/index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}

We then rerun our Webpack build:

/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ (__unused_webpack_module, __webpack_exports__, __webpack_require__) => {
  /* harmony export */ __webpack_require__.d(__webpack_exports__, {
    /* harmony export */ getUserName: () => /* binding */ getUserName,
    /* harmony export */
  });
  /* unused harmony exports getUserAccount, getUserPhoneNumber */

  const getUserName = () => "John Doe";

  const getUserPhoneNumber = () => "***********";

  const getUserAccount = () => userAccount;
  /***/
};

We see that the userAccount file has been removed from the bundle. We still see getUserAccount that references userAccount but this function has been marked by Webpack as dead code and it will be removed during minification.

The sideEffects flag is especially important for libraries that export their API through an index file that itself exports functions or variables from internal files. Without the side effects optimization, our bundlers would have to parse all the files where our exported variables are defined.

As noted by Webpack: "sideEffects is much more effective since it allows to skip whole modules/files and the complete subtree."

To better understand the difference between how both optimizations intervene:

sideEffects allows us to completely skip an imported module if none of its imported contents are used.
usedExports allows us to completely remove exports that are never imported by any module.

Ok but how is the result of skipping files different from just saying the exports of those files are unused?

Most of the time, tree shaking a library with and without the side effects optimization will give the same result. The same amount of code will be included in the final bundle. However, this is not true in some situations when analyzing the code for unused exports becomes too complex. The next part covers two of these situations where only the combination of small modules with the sideEffects optimization provides the best result.

To sum up this part:

Tree shaking consists of two parts: the used exports optimization and the side effects optimization
The side effects optimization is way more effective than just detecting unused export statements in every module
Make your library side effect free
Make sure to tell that your library is side effects free through the sideEffects property in the package.json file.

Preserve the library's module tree and split your code in small modules to fully benefit from the `sideEffects` optimization

You may have noticed that the example user-library we are using in this article is not bundled. The library just exposes a few JS files that I manually added.

Oftentimes, libraries will be bundled for multiple reasons:

Manage custom import paths
The library uses specific languages like SASS or TS that need to be transformed to CSS or JS for instance
The library needs to be available in multiple formats (ESM, CJS, IIFE ...)

Popular bundlers like Webpack, Rollup, Parcel or ESBuild are made to provide a bundle that can be served to a browser. They will thus have a tendency to create a single file regrouping all of your code so that only a single JS file needs to be sent through the network.

From a tree shaking perspective, this creates one problem: The side effects optimization is non existent as no modules can be skipped.

We are going to showcase two situations where splitted modules combined with the sideEffect optimization are essential for tree shaking.

A library module imports a CJS formatted dependency

To demonstrate the issue, we are going to bundle our library using Rollup. And we are going to have one of our library modules import a CJS formatted dependency: Lodash

rollup.config.js

export default {
  input: "src/index.js",
  output: {
    file: "dist/index.js",
    format: "esm",
  },
};

userAccount.js

import { isNil } from "lodash";

export const checkExistance = (variable) => !isNil(variable);

export const userAccount = {
  name: "user account",
};

Notice we are now exporting checkExistance and we import it in our library index file.

Here is the output in dist/index.js

import { isNil } from "lodash";

const checkExistance = (variable) => !isNil(variable);

const userAccount = {
  name: "user account",
};

const getUserAccount = () => {
  return userAccount;
};

const getUserPhoneNumber = () => "***********";

const getUserName = () => "John Doe";

export { checkExistance, getUserName, getUserPhoneNumber, getUserAccount };

Everything is bundled within a single file. Notice also that Lodash is imported at the top. We are still importing the same functions in our application meaning checkExistance is not used. However, after running Webpack, we see that the whole Lodash library is imported even though checkExistance is marked as unused:

/***/ "./node_modules/user-library/dist/index.js":
/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {

"use strict";
/* harmony export */ __webpack_require__.d(__webpack_exports__, {
/* harmony export */   "getUserName": () => (/* binding */ getUserName)
/* harmony export */ });
/* unused harmony exports checkExistance, userAccount, getUserPhoneNumber, getUserAccount */
/* harmony import */ var lodash__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(/*! lodash */ "./node_modules/user-library/node_modules/lodash/lodash.js");
/* harmony import */ var lodash__WEBPACK_IMPORTED_MODULE_0___default = /*#__PURE__*/__webpack_require__.n(lodash__WEBPACK_IMPORTED_MODULE_0__);

const checkExistance = (variable) => !isNil(variable);

const userAccount = {
  name: "user account",
};

const getUserPhoneNumber = {
	number: '***********'
};

const getUserAccount = () => {
	return userAccount
};

const getUserName = () => 'John Doe';

/***/ }),

/***/ "./node_modules/user-library/node_modules/lodash/lodash.js":
/*!*****************************************************************!*\
  !*** ./node_modules/user-library/node_modules/lodash/lodash.js ***!
  \*****************************************************************/
/***/ (function(module, exports, __webpack_require__) {

/* module decorator */ module = __webpack_require__.nmd(module);
var __WEBPACK_AMD_DEFINE_RESULT__;/**
 * @license
 * Lodash <https://lodash.com/>
 * Copyright OpenJS Foundation and other contributors <https://openjsf.org/>
 * Released under MIT license <https://lodash.com/license>
 * Based on Underscore.js 1.8.3 <http://underscorejs.org/LICENSE>
 * Copyright Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors
 */
// ...

Webpack is not able to tree shake Lodash because of its CJS format. This is a shame as we explicitly organized our library so Lodash would be imported only in the userAccount module that is unused in our application. If the module structure was preserved, Webpack would have detected that no userAccount exports were used and would simply have skipped the module and thus its Lodash import thanks to the sideEffects optimization.

In rollup, we can use the preserveModules option to preserve the module structure of our library.. Equivalents exist for other bundlers.

export default {
  input: "src/index.js",
  output: {
    dir: "dist",
    format: "esm",
    preserveModules: true,
  },
};

Rollup now keeps the original file structure. We can now run Webpack again:

/***/ "./node_modules/user-library/dist/index.js":
/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {

/* harmony export */ __webpack_require__.d(__webpack_exports__, {
/* harmony export */   "getUserName": () => (/* binding */ getUserName)
/* harmony export */ });
/* unused harmony export getUserAccount */

const getUserAccount = () => {
	return userAccount
};

const getUserName = () => 'John Doe';

/***/ })

Lodash is now skipped along with the userAccount module.

Code splitting

Preserving splitted module structure alongside the sideEffects optimization also benefits Webpack code splitting, a key performance optimization for big applications. Code splitting is widely used in web applications with multiple pages. Frameworks like Nuxt or Next both use page by page code splitting.

To illustrate the benefit, we will look at what happens when the library is bundled in a single file.

user-library/src/userAccount.js

export const userAccount = {
  name: "user account",
};

user-library/src/userPhoneNumber.js

export const userPhoneNumber = {
  number: "***********",
};

user-library/src/index.js

import { userAccount } from "./userAccount";
import { userPhoneNumber } from "./userPhoneNumber";

const getUserName = () => "John Doe";

export { userAccount, getUserName, userPhoneNumber };

In order to code split our user application, we will use the Webpack import syntax.

user-app/src/userService1.js

import { userAccount } from "user-library";

export const logUserAccount = () => {
  console.log(userAccount);
};

user-app/src/userService2.js

import { userPhoneNumber } from "user-library";

export const logUserPhoneNumber = () => {
  console.log(userPhoneNumber);
};

user-app/src/index.js

const main = async () => {
  const { logUserPhoneNumber } = await import("./userService2");
  const { logUserAccount } = await import("./userService1");

  logUserAccount();
  logUserPhoneNumber();
};

main();

The app bundle now has 3 files: main.js, src_userService1_js.main.js and src_userService2_js.main.js. Taking a closer look at src_userService2_js.main.js, we can see that the entire user-library bundle is added:

(self["webpackChunkuser_app"] = self["webpackChunkuser_app"] || []).push([
  ["src_userService1_js"],
  {
    /***/ "./node_modules/user-library/dist/index.js":
      /*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
      /***/ (
        __unused_webpack_module,
        __webpack_exports__,
        __webpack_require__
      ) => {
        "use strict";
        /* harmony export */ __webpack_require__.d(__webpack_exports__, {
          /* harmony export */ userAccount: () => /* binding */ userAccount,
          /* harmony export */ userPhoneNumber: () =>
            /* binding */ userPhoneNumber,
          /* harmony export */
        });
        /* unused harmony export getUserName */
        const userAccount = {
          name: "user account",
        };

        const userPhoneNumber = {
          number: "***********",
        };

        const getUserName = () => "John Doe";

        /***/
      },

    /***/ "./src/userService1.js":
      /*!*****************************!*\
  !*** ./src/userService1.js ***!
  \*****************************/
      /***/ (
        __unused_webpack_module,
        __webpack_exports__,
        __webpack_require__
      ) => {
        "use strict";
        __webpack_require__.r(__webpack_exports__);
        /* harmony export */ __webpack_require__.d(__webpack_exports__, {
          /* harmony export */ logUserAccount: () =>
            /* binding */ logUserAccount,
          /* harmony export */
        });
        /* harmony import */ var user_library__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(
          /*! user-library */ "./node_modules/user-library/dist/index.js"
        );

        const logUserAccount = () => {
          console.log(user_library__WEBPACK_IMPORTED_MODULE_0__.userAccount);
        };

        /***/
      },
  },
]);

userAccount is not marked as unused even though userService2 only uses userPhoneNumber... But why ?

We need to keep in mind that the usedExports optimization checks for used exports only within a module's scope. Only from there can Webpack remove unused code. From the perspective of our library module, both userAccount and userPhoneNumber are actually used. In this case, Webpack is not able to make a difference between the imports of userService1 and userService2 as seen on the following graph (both userAccount and userPhoneNumber are in green):

This means that Webpack is not able to tree shake the exports of each chunk independently when only relying on the usedExports optimization.

We now preserve our modules when bundling our library to allow for the sideEffects optimization:

Webpack still outputs the same 3 files but this time, src_userService2_js.main.js only contains the code coming from userPhoneNumber:

(self["webpackChunkuser_app"] = self["webpackChunkuser_app"] || []).push([
  ["src_userService2_js"],
  {
    /***/ "./node_modules/user-library/dist/userPhoneNumber.js":
      /*!***********************************************************!*\
  !*** ./node_modules/user-library/dist/userPhoneNumber.js ***!
  \***********************************************************/
      /***/ (
        __unused_webpack_module,
        __webpack_exports__,
        __webpack_require__
      ) => {
        "use strict";
        /* harmony export */ __webpack_require__.d(__webpack_exports__, {
          /* harmony export */ userPhoneNumber: () =>
            /* binding */ userPhoneNumber,
          /* harmony export */
        });
        const userPhoneNumber = {
          number: "***********",
        };

        /***/
      },

    /***/ "./src/userService2.js":
      /*!*****************************!*\
  !*** ./src/userService2.js ***!
  \*****************************/
      /***/ (
        __unused_webpack_module,
        __webpack_exports__,
        __webpack_require__
      ) => {
        "use strict";
        __webpack_require__.r(__webpack_exports__);
        /* harmony export */ __webpack_require__.d(__webpack_exports__, {
          /* harmony export */ logUserPhoneNumber: () =>
            /* binding */ logUserPhoneNumber,
          /* harmony export */
        });
        /* harmony import */ var user_library__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(
          /*! user-library */ "./node_modules/user-library/dist/userPhoneNumber.js"
        );

        const logUserPhoneNumber = () => {
          console.log(
            user_library__WEBPACK_IMPORTED_MODULE_0__.userPhoneNumber
          );
        };

        /***/
      },
  },
]);

src_userService1_js.main.js behaves the same way as it includes only the userAccount module from our library.

Looking at the graph, we still see that userAccount and userPhoneNumber are still considered as used exports as they are used at least once in our application. However, this time the sideEffects optimization is able to skip the userAccount module because it is never imported by userService2. The same thing happens for userPhoneNumber and userService1.

We can now understand that preserving the original module structure of the library is important. However, preserving this is useless if the original structure only has one module such as an index.js file with all the code inside. In order to make an optimal tree shakeable library, its code should be divided in multiple small modules that each handle one piece of the logic.

To use the tree analogy, one should see each leaf of the tree as a module. Smaller and weaker leafs will fall better when the tree is shaked! If the tree has fewer and bigger leafs, shaking it will not give the same result.

To sum up this part:

We should preserve the module structure of the library in order to fully benefit from the sideEffects optimization.
Libraries should be split in multiple small modules, each module exporting only one piece of logic
Tree shaking libraries in applications that use code splitting will only work with the sideEffects optimization.

Do not lose the module tree or the ES modules characteristics when transpiling your library

Bundlers are not the only tools that can harm the tree shaking of your library. Transpilers are also known to have an undesirable effect on tree shaking by removing ES modules and by not preserving the module tree.

One of the objectives of transpilers is to make your code compatible for browsers that do not necessarily support ES modules. We should remember though that our libraries are not meant to be served to browsers directly but should instead be consumed by applications. So transpiling our libraries to target specific browsers should be forbidden for two reasons:

When making a library, one does not know which browser should be targeted, only the application knows that.
Transpiling libraries can make them non tree shakeable

If the library needs to be transpiled for some reason, one needs to make sure the transpilation does not remove the ES module syntax or the original module tree for the same reasons explained in the last part of the article.

There are two tools that I know of that transpiled my libraries removing their tree shakeable characteristic.

Babel

Babel uses Babel preset-env to make your code compatible with one's target browsers. By default, this plugin will remove ES modules from the library. To make sure this does not happen, set the modules option to false:

module.exports = {
  env: {
    esm: {
      presets: [
        [
          "@babel/preset-env",
          {
            modules: false,
          },
        ],
      ],
    },
  },
};

Typescript

When compiling your code, typescript will transform your modules depending on the target and module options you set in the tsconfig.json file.

To make sure this does not happen, set the target and module options to at least ES2015 or ES6.

To sum up:

Make sure your transpilers/compilers do not remove the ES module syntax from your library bundle.
To check whether this happens, look at the library's output bundle and check for ESM import syntax.

Use the latest version of a tree shaking capable bundler

Tree shaking in javascript was popularized by Rollup. Webpack supports this since version 2 and bundlers keep getting better and better at optimizing tree shaking.

Remember when we talked about the innerGraph optimization that allows Webpack to link module exports to the module's imports? This optimization was introduced in Webpack 5. We have been using Webpack 5 in this article but it is important to note that this optimization is a game changer as it allows Webpack to recursively look for unused exports!

To show what it actually does, we can consider our index.js file in our user-library:

import { userAccount } from "./userAccount";

const getUserAccount = () => {
  return userAccount;
};

const getUserName = () => "John Doe";

export { getUserName, getUserAccount };

Our user-app only uses getUserName.

import { getUserName } from "user-library";

console.log(getUserName());

We can now compare the outputs with and without the innerGraph optimization. We are still using both the usedExports and sideEffects optimizations:

Without the innerGraph optimization (eg with Webpack 4):

/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/*! exports provided: userAccount, userPhoneNumber, getUserName, getUserAccount */
/*! exports used: getUserName */
/***/ (function(module, __webpack_exports__, __webpack_require__) {

"use strict";
/* harmony export (binding) */ __webpack_require__.d(__webpack_exports__, "a", function() { return getUserName; });
/* unused harmony export getUserAccount */
/* harmony import */ var _userAccount_js__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(/*! ./userAccount.js */ "./node_modules/user-library/dist/userAccount.js");

const getUserAccount = () => {
	return _userAccount_js__WEBPACK_IMPORTED_MODULE_0__[/* userAccount */ "a"]
};

const getUserName = () => 'John Doe';

/***/ }),

/***/ "./node_modules/user-library/dist/userAccount.js":
/*!*******************************************************!*\
  !*** ./node_modules/user-library/dist/userAccount.js ***!
  \*******************************************************/
/*! exports provided: userAccount */
/*! exports used: userAccount */
/***/ (function(module, __webpack_exports__, __webpack_require__) {

"use strict";
/* harmony export (binding) */ __webpack_require__.d(__webpack_exports__, "a", function() { return userAccount; });
const userAccount = {
	name: 'user account'
};

/***/ }),

With the innerGraph optimization (eg with Webpack 5):

/***/ "./node_modules/user-library/dist/index.js":
/*!*************************************************!*\
  !*** ./node_modules/user-library/dist/index.js ***!
  \*************************************************/
/***/ ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {

/* harmony export */ __webpack_require__.d(__webpack_exports__, {
/* harmony export */   "getUserName": () => (/* binding */ getUserName)
/* harmony export */ });
/* unused harmony export getUserAccount */

const getUserAccount = () => {
	return userAccount
};

const getUserName = () => 'John Doe';

/***/ })

While Webpack 5 is able to completely eliminate the userAccount module, this is not the case for Webpack 4 even though getUserAccount is marked as unused. This is because the innerGraph algorithm allows webpack 5 to link unused elements of our module with its imports. In our case, the userAccount module is used only by the getUserAccount function and can therefore be skipped.

This optimization does not work using Webpack 4. Developers should therefore be careful and limit the number of exports in a single file when using this version of Webpack. I a file contains multiple exports, Webpack will include all the file imports even though they may not be necessary for the desired export.

In general, we should make sure our bundlers are always up to date to benefit from their latest tree shaking optimizations.

Conclusion

Tree shaking a library is not something one just turns on by adding a specific line in a configuration file. Its quality depends on multiple factors and this article presents only a few of them. In the end though, whatever the issue might be, there are two important things we did in this article that can help anyone tree shake their libraries:

In order to see how well our library is tree shaked, we test it in a controlled environment using a bundler we know how to use.
We detect issues with our library setup not by looking at its configuration files but by inspecting its bundled output. This is what we have been doing all along in this article with our user-library and user-app examples.

I really hope this article helps you in your evergoing quest to make the best and most optimized libraries possible!

An introduction to DynamoDB data modeling

Mon, 12 Apr 2021 00:00:00 GMT

I discovered DynamoDB a few months ago, in the context of a very high traffic web application built using the Serverless Framework. DynamoDB - as well as Single-Table design - was fairly new to me, and I quickly noticed that there are a few concepts to grasp before designing the first data models. This article aims to provide a beginner-friendly tutorial for developers who are considering starting a project with DynamoDB, or who are just curious about this technology.

DynamoDB is a fully managed NoSQL database released by AWS in 2012. As the other main NoSQL solutions, such as MongoDB (2009) or Apache Cassandra (2008), it was designed at a time when storage prices were low compared to those of processing power. In the seventies, when relational databases were first introduced, this relationship was inverted. In particular, high storage prices gave incentives to limit storage needs by avoiding data duplication. You would design separate entities and put each one of them in a separate table, which drastically simplifies the data modeling exercise. You could then simply write the entity-relationship diagram (ERD), translate it into tables and retrieve the information of your choice by joining tables together.

With a NoSQL database such as DynamoDB, the join functionality does not exist, you're invited to put all your entities in the same table. This is the concept of Single-Table Design. By removing joins, you write more efficient queries, which reduces the needs of computing power. However, this comes at a cost: the data modeling exercise is slightly more complicated. The entity diagram does not give you the model anymore. The ways your application will interact with data - the access patterns - are now a key building block of your modelisation.

In this tutorial, I will take the example of a sample book review website. On the application, users can write a review for a book of their choice. They can also read the reviews written by others on any given book. Finally, users can read all the reviews written by a given user. As a bonus, we will talk about sorting these reviews in ante-chronological order.

Start from an entity-relationship diagram

The starting point of the exercise is the ERD. We won't need any joins here, which are the reflection of relationships between entities, but having these relationships in mind is necessary to know how you'll make it work without them.

In our case, the diagram is quite simple. There are 3 entities: books, users and reviews. There is a one-to-many relationship between users and reviews (a user can write several reviews), and another one-to-many relationship between reviews and books (a book can be reviewed several times). In this case, reviews implicitly represent the many-to-many relationship between users and books.

Entity-relationship diagram for users, books, and reviews.

As described on the diagram, we use the ISBN (International Standard Book Number) as a unique identifier for books, and the username for users. A review consists of a date, a grade, and a text.

Write down the access patterns

Before any attempt to model the keys in your table, you should write down the access patterns of your application. The modelisation's purpose is to handle all of them. It is important to be thorough in this step, or you might be unable to retrieve some information from your table. For the purpose of this exercise, let's consider three of them.

Access Pattern
Get all the reviews for a given book
Get all the reviews for a given user
Update a review

On the book page, we need the reviews for this particular book. On a user profile, we will query for the reviews written by this user. Finally, we will see how we would update a review.

Model the structure of the primary key

Once you have defined all of your access patterns, you can start the modelisation. The primary key is a core concept in DynamoDB, and almost the single piece of structure imposed to your table. It is thus at the core of the modeling exercise.

DynamoDB allows two kinds of primary keys:

Simple primary keys, made of a single element called partition key (PK).
Composite primary keys, made of partition key (PK) and sort key (SK).

The partition key is particularly important, as it represents an identifier of the server on which the data actually is (also called a storage node). Two items that have distinct partition keys will thus be difficult to query together.

The primary key related rules are simple:

A table must have a primary key
All the items in the table must have a value for that primary key
Each primary key must be unique <> an item is uniquely identified by its primary key
Each request to READ item(s) must specify the partition key *
Each request to WRITE an item must specify the full primary key

* Here I intentionally avoid scans, which are 9 times out of 10 bad practice

With these concepts in mind, you can start modeling. For each entity, we have to think about the best partition key and sort key to fulfill all the access patterns. In other words, the goal of this step is to fill the table below:

Entity	PK	SK
Book	`?`	`?`
Review	`?`	`?`
User	`?`	`?`

A common practice is to start with the top-level entity that has the highest number of relationships. Here, books and users play a somehow symmetric role so let's start with modeling books.

The goal is to be able to query at the same time for a book and its associated reviews. We thus have to make sure that the book reviews have the same partition key as the book itself, in order for them to be stored on the same physical machine. Using this partition key, DynamoDB will identify the associated storage node in a O(1) time. In DynamoDB terminology, we say that we put the book and its reviews in the same item collection.

Because an item in DynamoDB must be uniquely identified by its primary key, the sort key will be the way we differentiate a book from its reviews. Let's have a look at the following primary key structure:

Entity	PK	SK
Book	`BOOK#<ISBN Number>`	`BOOK#<ISBN Number>`
Review	`BOOK#<ISBN Number>`	`USER#<Username>`

Prefixing the key by the entity name is a common pattern. It gives us more clarity and compensates for the fact that we put all entities in the same table. The ISBN of the book is here used as a unique id. It enforces the uniqueness of a book's primary key. Finally, the sort key USER#<Username> allows us to differentiate reviews. It also identifies the author of the review. Implicitly, it says that a given user can not write two reviews about the same book (otherwise it violates the primary key's uniqueness principle).

Visualize your model in NoSQL Workbench

During the modeling exercise, it is good practice to model out some example data. It especially helps to visualize how a query would respond. The NoSQL Workbench is a great tool for that. At this stage of the modelisation, here is how a simple example would look like:

Main index - Books and reviews visualisation in NoSQL Workbench.

A row in this table is an item in DynamoDB. Both items share a common partition key, they are in the same item collection and can thus be queried together. We choose generic names for the key attributes (PK and SK) because they do not store the same type of information depending of the kind of entity they belong to.

To solve the first access pattern, just make a request for the items with a PK of BOOK#<ISBN>, with no condition on the sort key (remember that for a READ operation, specifying the sort key is optional). Both items in the table match this condition, so they will be returned together, sorted alphabetically on the sort key. In JSON format, you get something like

[
  {
    "PK": {
      "S": "BOOK#<ISBN>"
    },
    "SK": {
      "S": "BOOK#<ISBN>"
    },
    "Author": {
      "S": "J.R.R. Tolkien"
    },
    "Title": {
      "S": "The Fellowship of the Ring"
    },
    "PublicationYear": {
      "N": "1954"
    }
  },
  {
    "PK": {
      "S": "BOOK#<ISBN>"
    },
    "SK": {
      "S": "USER#<username>"
    },
    "Grade": {
      "N": "10"
    },
    "ReviewText": {
      "S": "Must read."
    },
    "ReviewDate": {
      "S": "2001-08-05T02:46:00"
    }
  }
]

Define secondary indexes if needed

Let's move on to the second access pattern. We want to be able to query for a user and all their reviews. We therefore need the user to be in the same partition as their reviews. But these already share a common partition key with the book they relate to. We thus somehow need the reviews to be in two distinct item collections. This is the notion of global secondary index (GSI).

When creating a GSI, you specify two attributes in your table that will act as a primary key in the secondary index. In other words, this index will be a copy of the data of the main table, in which the primary key is different. Using this, we can have the book and reviews to share a common PK in the main index, while having the reviews and the associated author in the same partition in a secondary index.

In our case, we can then adopt symmetric approaches in the main and secondary index. In the main index, the primary key structure will be like below:

Entity	PK	SK
Book	`BOOK#<ISBN Number>`	`BOOK#<ISBN Number>`
Review	`BOOK#<ISBN Number>`	`USER#<Username>`
User	`USER#<Username>`	`USER#<Username>`

See that the primary key structure for books and users follow the exact same pattern. In NoSQL Workbench, adding the review author to our example would bring us to:

Main index - Users, books and reviews visualisation.

Notice that both the user and their review have a SK of USER#<Username>. Having the SK of the main index as partition key in the secondary one would therefore solve the second access pattern. More precisely, we can just flip the PK and SK of the main index in the GSI. The secondary index in NoSQL Workbench would then be:

Secondary index - Users, books and reviews visualisation.

To solve the second access pattern, we make a request on the secondary index for the items with a partition key (here the SK attribute) of USER#<Username>, with no condition on the sort key. Both the user and the review match this condition, so they will be returned together in an array, sorted alphabetically on the sort key. To retrieve the user first, we can set the scanIndexForward property to false, so that DynamoDB reads the items in descending order. Note that by default (with scanIndexForward property set to true), DynamoDB will send the items in the order they are sorted in the table, which is the ascending order. If you are unsure, just look at your items in the NoSQL Workbench: top to bottom is the ascending ordering used by DynamoDB.

At this stage of the modelisation, most of the work is done. Inserting a review - the third access pattern - is trivial. We use the UpdateItem action of DynamoDB (you can learn more about writing items here), by specifying BOOK#<ISBN> for the partition key and USER#<Username> for the sort key (remember that for WRITE operations you must provide both the partition key and the sort key), along with the other attributes characterizing the review - grade, review text and date. By doing this, you implicitly specify a value for the PK and SK in the secondary index, and DynamoDB will thus take care of updating that item in the GSI as well.

What about sorting?

Sorting the query results before they are sent back to your application can be crucial in some situations. Going back to the book review example, imagine that you want to paginate the reviews for a particular book. More precisely, what if you want to show the 1 to X most recent reviews on the first page, then the X+1 to 2X most recent ones on the second page etc... Without sorting, you would have to retrieve all reviews on each page, and then get rid of those that are not in the appropriate range for that page. This would be very inefficient, especially if your application has a lot of contributors.

Let's cover how we would sort the reviews in ante-chronological order. As we have seen so far, in DynamoDB, almost everything is achieved using the primary key. It is also true for sorting. If you want a special ordering of the items returned from a query, there is only one way to do so: it has to be done with the sort key. Implicitly, it means that you need to arrange your items so that they are sorted in advance. This again stresses the need for thorough data modeling upfront.

In our case, the sort key thus needs to cary some sort of time indicator. Both unix timestamps (e.g. 744569696) or ISO-8601 date format (e.g. 2021-03-05T18:54:56+00:00) would work. The only requirement is that it must be sortable. Let's go for ISO- 8601 timestamps as they are human-readable. We can then prefix the sort key for reviews like so:

Main index - Timestamped version - Users, books and reviews visualisation.

For the review entity, the partition key and sort key in the secondary index are no longer an inversion of the primary key in the main table. That's why we have to add specific attributes to our table. We name them generically GSIPK and GSISK respectively as they do not store the same information per entity. They will act as partition and sort key in the secondary index.

To get a book and the associated reviews sorted, the query is the same as before. Query all the items with a partition key of BOOK#<Username> with no restrictions on the sort key. With the scanIndexForward property set to false, you would retrieve the book and then the review. As DynamoDB orders the items in ascending orders in your table, a more recent review would come after the first one:

Main index - Timestamped version - Sorted reviews.

With the scanIndexForward property set to false, you retrieve the items from bottom to top. This gives you the book and then the reviews from most recent to oldest. Finally, use the limit option (like in SQL) set to 16 for example and you restrict yourself to the 15 most recent reviews.

In the secondary index, the data is structured symmetrically. In this index, the item attributes GSI1PK and GSISK are the partition and sort key, respectively.

Secondary index - Timestamped version - Sorted reviews.

A query for items with a partition key (here the GSIPK attribute) of USER#<Username> with no restriction on the sort key gives us the user and their reviews. With scanIndexForward property set to false, the user will come first, and then their reviews in ante-chronological order. The response, in JSON format would be:

[
  {
    "PK": {
      "S": "USER#<Username>"
    },
    "SK": {
      "S": "USER#<Username>"
    },
    "MembershipDate": {
      "S": "2000-09-29T04:31:00"
    },
    "GSI1PK": {
      "S": "USER#<Username>"
    },
    "GSI1SK": {
      "S": "USER#<Username>"
    }
  },
  {
    "PK": {
      "S": "BOOK#<ISBN>"
    },
    "SK": {
      "S": "2002-08-05T02:46:00#USER#<Username>"
    },
    "Grade": {
      "N": "5"
    },
    "ReviewText": {
      "S": "Fantastic"
    },
    "ReviewDate": {
      "S": "2002-08-05T02:46:00"
    },
    "GSI1PK": {
      "S": "USER#<Username>"
    },
    "GSI1SK": {
      "S": "2002-08-05T02:46:00#BOOK#<ISBN>"
    }
  },
  {
    "PK": {
      "S": "BOOK#<ISBN>"
    },
    "SK": {
      "S": "2001-08-05T02:46:00#USER#<Username>"
    },
    "Grade": {
      "N": "5"
    },
    "ReviewText": {
      "S": "Must read."
    },
    "ReviewDate": {
      "S": "2001-08-05T02:46:00"
    },
    "GSI1PK": {
      "S": "USER#<Username>"
    },
    "GSI1SK": {
      "S": "2001-08-05T02:46:00#BOOK#<ISBN>"
    }
  },
]

Wrapping it all up

In this simple modeling exercise, we have set out the key steps that should be followed when creating a data model in DynamoDB. Write your entity diagram to identify the relationship between objects in your application. Carefully write down your access patterns and start modeling the structure of primary keys for your entities. Try to do your best with the main index, and then make the use of secondary indexes if needed. During the exercise, take a practical example and visualize it in the NoSQL Workbench.

Once done, celebrate, as scaling is not a problem anymore. In our example, a new book would represent a new partition, and therefore a new storage node: DynamoDB abstracts the complexity of database scaling from you. Because partitions are split up on different physical machines, and because the time to find the partition you query for is constant, your queries efficiency will remain stable as the amount of data increases.

However, this automatic scaling isn't free - the modelisation exercise is slightly more complicated than with relational database systems. You can't just go from your entities diagram to the data model. Access patterns must be thought of in advance, as they should drive the modelisation.

Because of this, there are a few situations in which DynamoDB might not be the optimal choice. If your application is meant to allow for a virtually unlimited number of access patterns - an analytics platform for instance - then DynamoDB will not be a good fit. Or, if you are building a new application from scratch, you may not know how its functionalities will look like in the longer run. It will then be tough to think of all the access patterns in advance. Finally, keep in mind the learning curve. If you plan to build a product as fast you can, and if your team is much more familiar with relational database systems, you might want to stick to it to maximize development speed.

If not in one of these situations, go and try DynamoDB. To speed up learning, I would strongly recommend reading The DynamoDB Book by Alex Debrie. You'll find plenty of well chosen examples as well as the main theoretical concepts behind it. Enjoy!

Icon Library in React: Why Inline SVG Are Better than a Font

Thu, 25 Mar 2021 00:00:00 GMT

How I managed to choose a performant icon system adapted to a Next.js project with styled-components.

Almost every website uses icons in order to quickly guide users or to improve the aesthetics of the page.

Usually, designers can export icons to SVG format. SVG stands for Scalable Vector Graphics which means they can grow indefinitely, as they are defined by a description, not by pixels like in pictures.

In a new Next.js/React project with a hundred icons, I had to choose the best way to integrate icons. In this article, I propose to:

Share a bench of four methods to integrate SVG
Compare the best promising methods: inline SVG and font
Detail the implementation of the retained method

So many ways to integrate SVG icons!

Once you have an SVG (let's say a wonderful arrow icon), there are at least four ways to import them... not all of these methods are worth it!

SVG as an image

An SVG can be used as the source of an img HTML tag:

const Icon = ({ src, ...imgProps }) => <img src={src} {...imgProps} />;

const ArrowIcon = <Icon src="icons/arrow.svg" alt="Big arrow" />;

Even if this icon can be manipulated exactly as an image, advantages of SVG, except file size reduction, cannot be used. In particular, customization is not possible. Moreover, if the file is moved or deleted, developers may not be aware of it. Let's see what's next!

SVG as a background image

A really simple way to create an icon is to set it as the background image of a div element.

import styled from "styled-components";

const Icon = styled.div`
  ${({ url }) => `background-image: url(${url});`}
  display: flex;
  align-items: center;
  justify-content: center;
  overflow: hidden;
  background-size: cover;
  background-position: center center;

  ${({ size }) => `
    width: ${size}px;
    height: ${size}px;
  `};
`;

const ArrowIcon = <Icon url="icons/arrow.svg" size={20} />;

The icon can be manipulated as a div element, so the size is customizable, but that's about it. Thus, let's go through another way.

SVG as font

One popular method to manage a big set of icons is to use a font. It is quite easy to generate a font with online tools such as IcoMooon or Fontello.

A generic icon component should look like:

import "./style.css";

const Icon = (props) => (
  <span>
    <i className={`icon-${props.slug}`} />
  </span>
);

Where the file style.css, normally created by a font generator, should be something like:

@font-face {
  font-family: "icons";
  src: url("fonts/icons.woff2") format("woff2"), url("fonts/icons.woff") format("woff");
  font-weight: normal;
  font-style: normal;
  font-display: block;
}

.icon-arrow:before {
  content: "\\e954"; /* `\\e954` is a character defined in the font */
}

For additional props, you can wrap the icon, replacing span by a styled component with props. For instance:

const WrappedIcon = styled.span`
  ${(props) => `
    i {
      ${props.color ? `color: ${props.color};` : ""}
      ${props.size ? `font-size: ${props.size}px;` : ""}
    }
  `};
`;

And finally get a simple icon component:

const ArrowIcon = <Icon slug="arrow" size={20} color="blue" />;

This method seems to be used in a lot of projects. Before challenging it, let's see the last one.

Inline SVG

Importing icons in React (granted it is properly configured, we'll see that later) is as simple as that:

import Arrow from "icons/arrow.svg";

const MyComponent = () => <Arrow />;

This is the method I chose in my case, let's understand why.

Comparison

Criteria ✅: OK ⚠️: Points of attention ⛔: KO if it is a regarded criterion	Inline SVG	Font	SVG as image	SVG as background image
Customization	✅ Highly customizable: `size`, `color`, etc. and specific properties `fill` and `stroke`	⚠️ Customizable like text only: `size`, `color`, etc.	⛔ Limited customization, same as image: `size`, etc.	⛔ Limited customization, via its div container: `size`, etc.
Customization	✅ All parts of the SVG can be customized finely	⚠️ Painful positioning as it uses text positioning CSS	⛔ Limited customization, same as image: `size`, etc.
Performance	✅ Smaller files compared to images	✅ Smaller files compared to images	✅ Smaller files compared to images	✅ Smaller files compared to images
	⚠️ Icons are loaded with the DOM: it may slow the page loading, but ensures all icons are loaded at the same time as the page	✅ Can be loaded thanks to a CDN	⚠️ Icons are loaded with the DOM: it may slow the page loading, but ensures all icons are loaded at the same time as the page	⚠️ Icons are loaded with the DOM: it may slow the page loading, but ensures all icons are loaded at the same time as the page
		⚠️ Font is loaded in parallel: blank characters can appear. All icons are loaded even if few are used
Maintainability	✅ Easy to add icons progressively	⚠️ Painful to regenerate a font, even if some external tools like Icomoon or Fontello can help	⛔ Difficult to assert images path are valid	⛔ Difficult to assert images path are valid
Developer Experience	✅ Good experience if SVG files are easy to customize	✅ Good experience once the font is configured	✅ Good experience	✅ Good experience
Accessibility	✅ Possibility to define `title` and `desc` tags	⛔ Not accessible without large efforts	✅ Accessible like images	⛔ Not accessible

Let's dive further into the comparison of the two concurrent methods: inline SVG vs. font...

Customization

Fonts are managed like text and it is therefore possible to customize an icon's size or color. Precise positioning is possible with the same properties as text (font-size, line-height, text-align, etc.), so it is quite painful when icons are used as buttons or images. For emojis used in a text, it could be more relevant, however.

SVG are customizable as images and properties fill and stroke allow to redefine colors respectively of the content or the border. All parts of the SVG can be separately customized, which is impossible to do with fonts.

➡️ Using inline SVG is the best choice if you need color variants of icons, or to customize precise parts of them.

➡️ Icons used as characters should be part of a font.

Performance

Fonts are loaded asynchronously, which means blank characters can appear while the font is being loaded.

All icons are loaded in the font, even if only a few of them are used on the page. However, it can be served by a CDN, a fast method if CORS is correctly configured.

On the contrary, SVG are loaded with the DOM, which avoids blank spaces, but relatively increases loading duration. An icon usually weighs from 500B to 5kB (less with SVG Optimization) so it can be non-negligible if hundreds of icons are used at a time - quite rare, however!

Fonts are also subject to anti-aliasing methods applied by browsers to font characters when scaling. SVG do not suffer from this issue and are fully scalable.

➡️ In pages with hundreds of icons, using a font reduces the page size. Serving it via a CDN is a performant option.

➡️ For pages with moderate use of icons, SVG is a better option to avoid anti-aliasing and blank characters at loading.

Maintainability

What if icons evolve along the development process?

With a font, you need to regenerate the font correctly, with the correct source files - otherwise, some character codes may change!

With SVG, you just need to add, replace or delete a file and update file imports in the project. In case of replacement, check the SVG structure is still the same, as fine customization of SVG parts may be affected.

➡️ If icons are likely to evolve, SVG files are a more consistent and independent format to handle icon addition, edition, or deletion.

Developer experience

Once icon systems are set, both methods are easy to use as developers, as seen in the first part.

The only point of attention can be when it is necessary to customize SVG icons that are already styled. For instance, with a fill or a width property already set:

<svg width="15px">
  <path fill="none" d="..." />
</svg>

In this case, applying custom fill or width won't be applied. Cleaning the SVG by removing fill="none" and width="15px" should fix the issue.

➡️ Once an icon system is set, the developer experience should be satisfying in both cases.

Browser compatibility

Fonts are supported by almost all browsers, using the @font-face rule. All modern web browsers support WOFF2 and WOFF which are optimized formats for the web, but other formats can be used for broader browser support. However, Opera Mini does not support @font-face at all.

All modern browsers now support inline SVG method, even Internet Explorer which only does not support CSS transforms (more details on caniuse website).

➡️ Both methods are supported by more than 98% of web browsers. Inline SVG is slightly better supported and has better fallback options.

Accessibility

Serving accessible fonts is hard. If you are adventurous, this article details the tricks to guarantee accessible fonts, so I won't expand on it here.

If accessibility is important for your project, then SVG are made for this. An SVG file may have a title or even a desc (for description) tags. For better browser support, it is possible to add an id to these tags and refer to them the attributesaria-labelledby as the following minimal example shows.

<svg aria-labelledby="circleTitle circleDescription" role="img">
  <title id="circleTitle">A circle</title>
  <desc id="circleDescription">A red circle with a blue border</desc>
  <circle cx="50" cy="50" r="40" stroke="blue" stroke-width="2" fill="red" />
</svg>

For more details, I advise you to look at this CSS-tricks page.

➡️ For accessible projects, SVG is the best option with no hesitation.

A winner?

A font system is an option to consider if your project uses a huge amount of icons per page, is focused on performance, does not need to be accessible, and has a graphic chart that won't evolve.

In other cases, for better flexibility and user experience, I would recommend an inline SVG system. That's what I chose in the case of a new B2C project with common use of icons (<30 per page) and a library with possible evolutions.

In the following part, I will explain how to configure such an icon system.

Configuring an SVG icon system with Next.js, React and styled-components

Here are the four simple steps I followed to integrate icons in a Next.js/React project using styled-components as styling library.

Install babel-plugin-inline-react-svg:

# With npm
npm install --save-dev babel-plugin-inline-react-svg
# or, with yarn
yarn add --dev babel-plugin-inline-react-svg

This package will allow the import of SVG files as React components. It also uses SVGO to optimize SVGs.

Note: SVGR also offers good alternatives to babel-plugin-inline-react-svg depending on your bundler, such as @svgr/webpack.

Add inline-react-svg plugin to babel configuration (babel.config.js or .babelrc):

plugins: ["inline-react-svg"];

For further configuration, refer to the npm page.

In a TypeScript project, you should also add a module declaration in a type file:

// fileTypes.d.ts

declare module "*.svg" {
  import React = require("react");
  const src: (props: React.SVGProps<SVGSVGElement>) => JSX.Element;
  export default src;
}

Now, it is possible to import directly SVG as React components. For a resilient icon system, it is possible to create a JSX/TSX file per icon:

// icons/Arrow.jsx
import Arrow from "./arrow.svg";

// ... (possible customization of the component)

export default Arrow;

With this method, refactoring icons is easier if the import method changes.

Warning: using a unique index file with all SVG icons gathered in it is not recommended, because it prevents from tree-shaking the library.

Import SVG files as React components:

import Arrow from "icons/Arrow";

const MyComponent = () => <Arrow />;

It is also possible to customize icons with styled-components, as shown in this example:

// MyComponent.tsx
import styled from "styled-components";
import Arrow from "icons/Arrow";

const CustomSpan = styled.span`
  color: red;
  font-size: 40px;
`;

const CustomArrow = styled(Arrow)`
  width: auto;
  height: 30px;
  stroke: blue;
  stroke-width: 50px;
  fill: currentColor; /* fill with the current color, here red */
`;

const MyComponent = () => (
  <button>
    <CustomSpan>
      Go <CustomArrow />
    </CustomSpan>
  </button>
);

...which renders as a wonderful button:

That's all you need to manage icons in your project!

File Download with Token Authentication and Javascript

Wed, 24 Mar 2021 00:00:00 GMT

Recently on a project of mine, I was presented with a problem: I needed to allow our users to download a ZIP archive of files from our back-end, based on complex authorisation rules, following a four-step process:

The front-end would call the back-end with a list of file names to download.
The back-end would authenticate the user with their JWT token.
It would then check if the user was allowed to retrieve those files.
Finally, it would send them the archive back.

Up until now, we were relying on ‘security through obscurity’, meaning we were trusting the uniquely generated file names to prevent users from snooping around and downloading files they were not supposed to retrieve.

But this was not enough: how could we revoke the rights of the user on a file? How could we be certain that a user would not be accessing a file they were not supposed to access? We needed to add authentication to the route.

The Unauthenticated Case

First, let me show you the code we were using until now. In this case, the back-end is a Spring Boot microservice, and the front-end is an Angular SPA.

Below is a simplified version of the unauthenticated case. What this does is define a GET route /files/download taking a set of file IDs as parameters, and retrieving, zipping and sending the files back to the client.

@GetMapping("/files/download")
public ResponseEntity<ByteArrayResource> download(
    @RequestParam Set<UUID> fileIds
) {
    ByteArrayResource zippedFiles = new ByteArrayResource(
        fileService.getZippedFiles(fileIds)
    );

    return ResponseEntity
        .ok()
        .contentLength(zippedFiles.contentLength())
        .header(HttpHeaders.CONTENT_TYPE, "application/zip")
        .header(
            HttpHeaders.CONTENT_DISPOSITION,
            ContentDisposition
                .builder("attachment")
                .filename("archive.zip")
                .build()
                .toString()
        )
        .body(zippedFiles)
    ;
}

So now, if you make a GET request to /files/download?fileIds=UUID, you’ll receive the file with name UUID zipped. This allows you to keep your front-end code extremely simple because a simple anchor element pointing to the URL to the back-end route will be enough to trigger a download pop-up in the client’s browser, thanks to the Content-Disposition header.

So your front-end component could retrieve the required file names, concatenate them somehow, and then simply display a link to the back-end URL, like so:

<a
  href="{{ backendUrl }}/files/download?fileIds={{ fileNames$ | async }}"
>Download your archive</a>

This is all good and well, but when you do this, you cannot forward JWT tokens, so you cannot authenticate your user: you need to download the files some other way.

Adding Authentication to the Route

So first, you need to add some kind of authentication and authorization. Though my project required complex authorization rules, let us say you only want to be sure the user is logged in before sending them the files. This is only a matter of adding a decorator to your back-end route:

@GetMapping("/files/download")
@PreAuthorize("hasRole(T(fr.theodo.security.Roles).USER)")
public ResponseEntity<ByteArrayResource> download(…) {…}

This is the easy part. Now, if you were to attempt using the aforementioned link, you would get an error: as no authentication token is provided, the back-end responds with a 401 Unauthorized response. How do we fix that?

Retrieving the File with JavaScript

Downloading the file will be done in two steps: first, you will download the file using JavaScript, allowing you to set the authentication token, then, you will ‘forward’ the file to your user.

Alternatively, note that you could set up a different authentication method specifically for this route, such as a cookie-based one, though this would mean having two different authentication methods on your project, with one being used by a single route… so I would advise against it.

Downloading the File

Assuming you already perform authenticated calls to your back-end using some kind of API client, downloading the file will be straightforward: you will instantiate your client in your component, and call your back-end (here, it is done with this.apiClient.downloadZipFile).

The switchMap RxJS operator allows you to discard any previous calls if this.project$ changes; you can read more about it on Learn RxJS.

export class MainPanelComponent {
  @Input() project$!: Observable<Project>;

  constructor(private apiClient: ApiClientService) {}

  private getZip(): Observable<string> {
    return this.project$.pipe(
      switchMap(project => this.apiClient.downloadZipFile(project.files)),
    );
  }
}

‘Forwarding’ the File to Your User

Now that you have retrieved your file, returned as an Observable by switchMap, you need to trigger the save on your user’s computer. This is only a matter of chaining two new operators in the RxJS pipe:

…
return this.project$.pipe(
  switchMap(project => this.apiClient.downloadZipFile(project.files)),
  map(data => window.URL.createObjectURL(data)),
  tap(url => {
    window.open(url, '_blank');
    window.URL.revokeObjectURL(url);
  })
);
…

First, you map the retrieved data to an object URL (a simple string). MDN will explain them in greater details, but the main things you need to know are that they are a way to create local URLs pointing to the browser memory… and that they are not supported by Internet Explorer, so you may want to keep that in mind depending on your target audience.

Finally, you tap into this new Observable to expose the file to your user (simply opening a new window with this local URL will work) and clean up behind yourself by removing the object URL (this allows the browser to free up some memory once the file is downloaded).

Triggering the Download

You may have noticed that for now, you have a getZip method, but you do not call it. To remedy this, you only need two new lines of code in your component and one in its template:

export class MainPanelComponent {
  @Input() project$!: Observable<Project>;

  zipDownloadTrigger = new EventEmitter<void>();

  constructor(private apiClient: ApiClientService) {
    this.zipDownloadTrigger.pipe(exhaustMap(this.getZip)).subscribe();
  }

  …
}

<button (click)="zipDownloadEmitter.emit()">Download your archive</button>

This defines an event emitter, which emits after each click on a button. You chain this emitter with an exhaustMap RxJS operator, which will call the getZip method, and wait for it to complete before resuming listening to the event emitter (hence the ‘exhaust’; again, refer to Learn RxJS for more details).

This means that all clicks on the button are ignored while the component is downloading the file. Note that you could do without this, in which case you could simply pass the getZip method as a click callback on the button.

Note that for the sake of simplicity, I use a button here, but this is a bad practice in terms of accessibility: you should always use a download link to download a file. Here you could have a button to trigger the download, then display the link with the local URL as its href attribute.

UX Niceties and the Final Code

If you put all this together and add a simple loading toggle, you get the following code:

export class MainPanelComponent {
  @Input() project$!: Observable<Project>;

  isTheArchiveLoading: boolean = false;
  zipDownloadTrigger = new EventEmitter<void>();

  constructor(private apiClient: ApiClientService) {
    this.zipDownloadTrigger.pipe(exhaustMap(this.getZip)).subscribe();
  }

  private getZip(): Observable<string> {
    this.isTheArchiveLoading = true;

    return this.project$.pipe(
      switchMap(project =>
        this.apiClient
          .downloadZipFile(project.files)
          .pipe(finalize(() => { this.isTheArchiveLoading = false; }))
      ),
      map(data => window.URL.createObjectURL(data)),
      tap(url => {
        window.open(url, '_blank');
        window.URL.revokeObjectURL(url);
      })
    );
  }
}

This ensures that you can show a loader to your user if isTheArchiveLoading is true. The finalize operator in the pipe on downloadZipFile is called when downloadZipFile resolves, much like the Promise.finally() method.

And here you go, you have a fully functional downloading method, with an authenticated route, called from your front-end!

Improve accessibility on your Angular Apps with Jasmine and Axe

Mon, 22 Mar 2021 00:00:00 GMT

At Theodo, we have multiple projects using Angular and we are currently focusing on building more accessible websites. (learn more about a11y)

We had an experience with Axe and jest-axe on other React projects, and we wanted something similar for Jasmine, the test framework on the Angular project.

What is Axe

Axe is an open source accessibility testing engine by Deque. It can automatically find about 30% of accessibility defects, such as missing labels on inputs, missing alt on images, ...

We found some tests examples with Jasmine in the Axe documentation but it did not seem as straightforward to use as jest-axe. So we decided to solve this problem and make an open source matcher for Jasmine

How to use jasmine-axe

Here is an example on how to use jasmine-axe:

import { TestBed } from '@angular/core/testing';
import { FormsModule } from '@angular/forms';
import { axe, toHaveNoViolations } from 'jasmine-axe';
import { FormComponent } from './form.component';

describe('FormComponent', () => {
  beforeEach(() => {
    TestBed.configureTestingModule({
      imports: [FormsModule],
      declarations: [FormComponent],
    });
    TestBed.compileComponents();
    jasmine.addMatchers(toHaveNoViolations);
  });

  it('should pass accessibility test', async () => {
    const fixture = TestBed.createComponent(FormComponent);

    expect(await axe(fixture.nativeElement)).toHaveNoViolations();
  });
});

Axe will find the violations and suggest solutions to fix them.

A violation was found in a component: Axe suggests some solutions to fix it.

We cannot rely only on those tests for ensuring full accessibility of a website, but it allows us to automate some checks and catch common errors.

We are eager for feedback so try it on your Angular project and tell us if it helped your team improve accessibility!

Theodo

Building Terravision(2021) using Deck.gl and Carto

Intro

Case Study

The Data

Using Deck.gl with next.js and react

Technical Deep Dive

The Power of the_geom

More tiles than the kitchen floor

The End...?

Comparison of Cloud Run and Lambda to render Blender scenes serverlessly

Test protocol

vCPUs and RAM

Container image

Region

Execution environment

Test scene

Results

Next steps

How to Generate Beautiful PDFs with React and Puppeteer

Render a printable version of your page

From dynamic webpage to static rendering

Deal with page breaks and fixed size with CSS

How to use Puppeteer to generate your PDFs

How Puppeteer works?

How to use it?

Add some options to customize the PDF

Conclusion

A Quick Intro to Elm for React Developers

Why would a React developer learn Elm?

An Example Project

JavaScript:

Elm:

What benefits does this style provide?

Where are these famous types?

What next?

Migrating from React Redux to React Query

Our Initial Project

Converting the Redux Store to React Query

Converting the Saga to a Mutation

Final Project

How to Create a Localized Date Pipe in Angular 📅

Use a pipe in the template

Different ways to translate dates in a template

Approach 1: Setting the default locale

Approach 2: Passing the locale as arguments of the date pipe

Approach 3: Using a custom pipe localizedDate

Our own custom localized date pipe

Pure vs Impure pipe

A Stackblitz is worth a thousand words

The last word

How I Made My Life Easier by Letting QA Write E2E Tests?

Testing the application - how to do it effectively?

What if QA could write tests?

Writing tests without coding (or very little)

Make our cucumber tests run

How was this solution for us?

And if you want to have your QA writing the e2e tests?

Architecture guidelines for large Angular applications

TLDR;

Old habits that get you wrong

Those patterns you need to let go of

Improving readability in consulting companies

Guidelines for clean and efficient Angular coding

Use angular-cli

You are writing presentation code in .ts or .html ? Make a pipe

Make reactive Services that support your components business logic

Code in a reactive way with observables as inputs and funtions as outputs

Split your services when they become bigger than 100 lines

When your base module becomes too big and handles many topics : split into smaller modules

Let's study an example: Breaking down one component into smaller simpler pieces

Avoid Input/Output and prefer injecting specialized services

What are the advantages of doing so ?

AWS Fargate: harness the power of serverless for long-running computational tasks

The mission

The challenge

The strategy

Dockerize the tasks: Fargate + ECR + VPC

Syncing input and output directories: S3

Data model: DynamoDB

Approach 2: Passing the locale as arguments of the `date` pipe

Approach 3: Using a custom pipe `localizedDate`