Theodo

Build your own stickers component with react-konva

Mon, 16 Nov 2020 00:00:00 GMT

Konva.js is an awesome HTML5 Canvas JavaScript framework that enables high performance animations for desktop and mobile applications.

The framework's capabilities are useful and versatile - it can be integrated for many different and varied use cases. Konva.js deals well with touch genstures, drag and drop and animations to name just a few!

To see the breadth of the konva.js framework see the konva.js demos page.

React-Konva

React-Konva allows us to work with both React applications and Konva.js together seamlessly.

Here we'll create draggable stickers which can be deleted, suitable for both desktop and mobile applications. There are great react-native stickers libraries out there, however there really aren't many libraries to implement it in a react app. Therefore, react-konva is a great solution!

Show me the code!

Check out my code sandbox for the demo here!

Let's get started...

Adding a konva image

Let's take this tutorial through step-by-step. We need a canvas to add our stickers to. We'll add an image here to replicate a selfie, the sticker's most common use case.

import React from "react";
import { Image as KonvaImage, Layer, Stage } from "react-konva";
import useImage from "use-image";

export default function App() {
  const [image] = useImage("example-image.png");

  return (
    <Stage width={500} height={500}>
      <Layer>
        <KonvaImage image={image} />
      </Layer>
    </Stage>
  );
}

What is going on here?

The useImage hook handles loading the image
The Stage acts as our canvas
The Layer is tied to the canvas element and can contain Groups, Shapes and Images
The KonvaImage acts like an img tag and allows us to display our image

Photo by Moose Photos from Pexels

Aside: When I was building this for a photobooth experience I utilised the react-html5-camera-photo library to take the photo that we used as the image in the canvas, but let's stick to the stickers for now 👀

Creating the sticker button

Let's add a button with our desired sticker. We'll click this button whenever we want to add a new sticker to the canvas.

<button
    className="button"
    onMouseDown={() => {
      addStickerToPanel({
        src: "sticker-1.png",
        width: 100,
        x: 100,
        y: 100
      });
    }}
>
  <img alt="Eiffel Tower" src="sticker-1.png" width="100px" />
</button>

We use the button's onMouseDown property to add the sticker to our array of images to be added to our canvas. We set the width and the x and y position where the sticker should appear on the canvas.

We initialise an empty array called images, using react's useState hook, and add the width and src properties to the array when we click a sticker!

const [images, setImages] = useState([]);

  const addStickerToPanel = ({ src, width, x, y }) => {
    setImages((currentImages) => [
      ...currentImages,
      {
        width,
        src,
        x,
        y
      }
    ]);
  };

Adding a draggable sticker

Let's create a separate component that contains our individual stickers which will be on the canvas.

We can only use konva components within our Stage in App.js so we use a Konva Image component.

import useImage from "use-image";
import React from "react";
import { Image as KonvaImage } from "react-konva";

export const IndividualSticker = ({ image }) => {

  const [stickerImage] = useImage(image.src);
  const stickerHeight = stickerImage
    ? (image.width * stickerImage.height) / stickerImage.width
    : 0;

  return (
    <KonvaImage
	  draggable
      width={image.width}
      height={stickerHeight}
      image={stickerImage}
      x={image.x}
      y={image.x}
    />
  );
};

Again, we load our image using the useImage hook
To get the image to scale properly we add the stickerHeight calculation
In the KonvaImage we add the x and y properties to determine where the sticker appears on the canvas when the button is clicked.
The draggable property allows us to drag the image within the canvas!

In App.js let's map over the images within our layer.

{images.map((image, i) => {
  return <IndividualSticker key={i} image={image} />;
})}

Note: The sticker will always appear on top of the image if the stickers are below the image in our file (so there's no need for a z-index).

Multiple Stickers

Let's add multiple different styles of stickers 🤩

Create a stickers data file to include an array of your chosen stickers. Include the url, width and alt name. Add as many as you like!

export const stickersData = [
  {
    url: "sticker-1.png",
    width: 100,
    alt: "Eiffel Tower"
  },
  {
    url: "sticker-2.png",
    width: 150,
    alt: "Statue of Liberty"
  },
  {
    url: "sticker-3.png",
    width: 60,
    alt: "Big Ben"
  }
];

In App.js import the stickers data and map over them to add all the buttons!

{stickersData.map((sticker) => {
        return (
          <button
            className="button"
            onMouseDown={() => {
              addStickerToPanel({
                src: sticker.url,
                width: sticker.width,
								x: 100,
								y: 100
              });
            }}
          >
            <img alt={sticker.alt} src={sticker.url} width={sticker.width} />
          </button>
        );
      })}

Checkpoint: Draggable stickers that work well on mobile and desktop! 🎉

⁉️ WAIT, but I want to be able to delete the stickers!?

✅ Okay, let's continue with the tutorial. Things get a little more complicated here!

Deleting the stickers

In order to delete stickers from the canvas we add a cross icon to the top right corner of each sticker.

On mobile the cross appears on long tap of the sticker
On desktop the cross appears when you hover over the sticker

Let's add another KonvaImage to our individual sticker and wrap it in a Group so the sticker and cross icon are grouped.

export const IndividualSticker = ({ image, onDelete, onDragEnd }) => {
  const [stickerImage] = useImage(image.src);
  const [deleteImage] = useImage("cancel.svg");

  const stickerWidth = image.width;
  const stickerHeight = stickerImage
    ? (image.width * stickerImage.height) / stickerImage.width
    : 0;
  return (
    <Group
      draggable
      x={image.x}
      y={image.y}
      onDragEnd={(event) => {
        onDragEnd(event);
      }}
    >
      <KonvaImage
        width={image.width}
        height={stickerHeight}
        image={stickerImage}
      />
      <KonvaImage
        onTouchStart={onDelete}
        onClick={onDelete}
        image={deleteImage}
        width={25}
        height={25}
        offsetX={-stickerWidth / 2 - 20}
      />
    </Group>
  );
};

Render a cross icon using the useImage hook
Add the draggable, x and y properties to the Group so that they are common to both images
Use the offset properties on the delete button to position the cross where you like
Add the onClick property and pass the onDelete function. Also add the onTouchStart property so that this works on mobile too!
Pass an onDelete function detailed in App.js
- Use a splice to remove the image from the local state when the delete button is tapped/clicked

onDelete={() => {
  const newImages = [...images];
  images.splice(i, 1);
  setImages(newImages);
}}

We need to ensure that the x and y position of each sticker in the local state is updated appropriately - we make use of the Group's onDragEnd property. We pass in the following function from the App.js file. This updates the x and y position of the sticker at the end of the drag movement.

onDragEnd={(event) => {
    image.x = event.target.x();
    image.y = event.target.y();
  }}

Checkpoint: The cross shows when we add a sticker to the photo. When we tap/click the icon, it deletes the sticker as expected

Now, we only want to show the cross when we hover on desktop or long press on mobile - let's continue!

We use the react-use useLongPress and useHoverDirty to show the cross on long tap (mobile) or on hover (desktop).

Let's amend IndividualSticker.tsx to look like the following...

const imageRef = useRef(null);
const isHovered = useHoverDirty(imageRef);

const [showDeleteButton, setShowDeleteButton] = useState(false);
const [isDragging, setIsDragging] = useState(false);

const onLongPress = () => {
    setShowDeleteButton(true);
  };
const longPressEvent = useLongPress(onLongPress, { delay: 200 });

useEffect(() => {
    if (isHovered) {
      setShowDeleteButton(true);
    } else {
      setTimeout(() => {
        setShowDeleteButton(false);
      }, 2000);
    }
  }, [isHovered]);

return (
    <Group
      draggable
      x={image.x}
      y={image.y}
      onDragStart={() => setIsDragging(true)}
      onDragEnd={(event) => {
        setIsDragging(false);
        onDragEnd(event);
      }}
    >
      <KonvaImage
        ref={imageRef}
        width={image.width}
        height={stickerHeight}
        image={stickerImage}
        {...longPressEvent}
      />
      {showDeleteButton && !isDragging && (
        <KonvaImage
          onTouchStart={onDelete}
          onClick={onDelete}
          image={deleteImage}
          width={25}
          height={25}
          offsetX={-stickerWidth / 2 - 20}
        />
      )}
    </Group>
  );

We add a ref to the image that will be hovered. useHoverDirty is true when the image is hovered
We store the isDragging and showDeleteButton booleans in the local state using the useState hook
When we long tap and image we set the showDeleteButton state to true after a delay of 200ms
The useEffect ensures that the delete button disappears after the 2s of the image being hovered if it is not deleted.
We only display the delete button if the the showDelete button state is true and isDragging is false. This ensures that we don't show the delete button when we're dragging. This is a little hacky but it considers all the cases for mobile and desktop use!

We now need to ensure that the cross disappears if a user taps an area not associated with the sticker.

In App.js

const addStickerToPanel = ({ src, width, x, y }) => {
    setImages((currentImages) => [
      ...currentImages,
      {
        width,
        x,
        y,
        src,
        resetButtonRef: createRef()
      }
    ]);
  };

const resetAllButtons = useCallback(() => {
    images.forEach((image) => {
      if (image.resetButtonRef.current) {
        image.resetButtonRef.current();
      }
    });
  }, [images]);

  const handleCanvasClick = useCallback(
    (event) => {
      if (event.target.attrs.id === "backgroundImage") {
        resetAllButtons();
      }
    },
    [resetAllButtons]
  );

We create a new ref on each button using React's createRef
We add the handleCanvasClick which resets all the refs on the buttons so that we don't display the cross!

In the Stage add the onClick and onTap to handle this.

<Stage
        width={600}
        height={400}
        onClick={handleCanvasClick}
        onTap={handleCanvasClick}
      >

The Result

Using react-konva to build a great photobooth experience within a react app was really fun and I'll look forward to using this framework again! Check out more react-konva examples here and don't forget to checkout my codesandbox!

Sources + links:

Create resizeable split panels in React

Mon, 09 Nov 2020 00:00:00 GMT

Resizable containers empower different users to customize a UI to emphasise what they find most important. A basic implementation of this is a split view or split pane, allowing users to enlarge content that is more relevant to them.

There are libraries that can implement this feature for you, but creating it yourself gives you full control over the styling and functionality, so you can rapidly tailor it to your project. The full feature can be done in under 150 lines, with no dependencies, and in my case it was faster to add to my project than using a third-party library. It's also a component that covers a wide range of the features React has to offer, which is great for quickly getting to grips with what the library can do.

You can find the finished component right here.

(Credit to Seif Ghezala for the initial inspiration I've expanded on in this article)

Getting started

Creating the component

Let's start out by creating a functional component. We want to be able to pass in the content to be displayed in each panel, so let's go ahead and pass in the LeftContent and RightContent as props, and wrap them in styled div so they're displayed side by side.

import React from "react";

interface SplitViewProps {
  left: React.ReactElement;
  right: React.ReactElement;
}

export const SplitView: React.FunctionComponent<SplitViewProps> = ({
  left,
  right,
}) => {
  return (
    <div className="splitView">
      {left}
      {right}
    </div>
  );
};

.splitView {
  height: 100%;
  display: flex;
  flex-direction: row;
  align-items: flex-start;
}

Now let's add the divider. I'm going to keep things simple and just use a div with a margin. The width will be fixed at a few pixels, and I want the height to grow with the content in the panels either side, which we can do with height 100%.

  return (
    <div className="splitView">
      {left}
      <div className="divider"/>
      {right}
    </div>
  );

.divider {
  width: 2px;
  height: 100%;
  margin: 1rem;
  border: 2px solid #808080;
}

Alright here's our basic component set up.

Now for the fun part!

Time to start resizing

So the end goal is to click and drag the divider and resize both panels accordingly. The trick here is to resize just one of the panels, and then let flex-grow and flex-shrink take care of the other.

Let's control the panel on the left. We're going to need a way of storing and setting its width so we can create some state for our component using the useState hook.

const [leftWidth, setLeftWidth] = useState<undefined | number>(undefined);

The panel on the left is going to have quite a bit of its own functionality so lets move that div into its own new component called LeftPanel. We want it to use the state we just created so lets pass in leftWidth and the setter too.

const LeftPanel: React.FunctionComponent<{
  leftWidth: number | undefined;
  setLeftWidth: (value: number) => void;
}> = ({ children, leftWidth, setLeftWidth }) => {

  return <div>{children}</div>;
};

Time for the magic. The ref!

React refs allow us to access the properties of elements in the DOM. If we use it on the div around our left panel content, we'll be able to both get and set the width of it directly.

To use refs, first create one using React's createRef function. Then attach it to the desired element using the ref property.

  const leftRef = createRef<HTMLDivElement>();
  return <div ref={leftRef}>{children}</div>;

By doing this, the properties of that element are now accessible under leftRef.current.

Time to connect our state and the ref with another React hook, useEffect. First we want to make sure that if our state isn't defined it gets initialized to the width of the left panel.

Then we want to make sure that whenever our state changes, we change the width of the left panel too. We can access the width using current.clientWidth.

const LeftPanel: React.FunctionComponent<{
  leftWidth: number | undefined;
  setLeftWidth: (value: number) => void;
}> = ({ children, leftWidth, setLeftWidth }) => {
  const leftRef = createRef<HTMLDivElement>();

  React.useEffect(() => {
    if (leftRef.current) {
      if (!leftWidth) {
        setLeftWidth(leftRef.current?.clientWidth);
        return;
      }

      leftRef.current.style.width = `${leftWidth}px`;
    }
  }, [leftRef, leftWidth, setLeftWidth]);

  return <div ref={leftRef}>{children}</div>;
};

Now whenever we change the leftWidth state in our original SplitView component, it will change the width of the left panel!

Notice the array after the contents of the useEffect. That dependency array makes sure that the code in the effect runs whenever either of those values change.

So we could do the same thing for the right panel, but as I said before, we can make things much simpler by giving it flex-grow (here I've used the shorthand flex: 1).

export const SplitView: React.FunctionComponent<SplitViewProps> = ({ left, right}) => {
  const [leftWidth, setLeftWidth] = useState<undefined | number>(undefined);
  return (
    <div className="splitView">
      <LeftPanel leftWidth={leftWidth} setLeftWidth={setLeftWidth}>
        {left}
      </LeftPanel>
      <div className="divider"/>
      <div className="rightPane">{right}</div>
    </div>
  );
};

.rightPane {
  flex: 1;
}

Now wasn't that easier?

We're all set to resize both of our panels. We just need something to trigger it.

Getting the mouse involved

We want to resize our panels when we click and drag the separator. We can break that down into 3 stages:

Clicking down on the separator
Dragging the mouse
Releasing the separator

Conveniently, we can capture each of these stages with 3 mouse events;

onMouseDown: Set a flag to start tracking mouse movements. Store the initial position of the mouse at this moment.
onMouseMove: If the flag is set, track the mouse's horizontal position, and resize the left panel accordingly
onMouseUp: Clear the flag

Let's start with onMouseDown. When the user clicks on the separator, we want to store the mouse's horizontal position at that moment, and set a flag to start tracking any horizontal movement after that. Sounds like we need some more useState hooks.

  const [separatorXPosition, setSeparatorXPosition] = useState<undefined | number>(undefined);
  const [dragging, setDragging] = useState(false);

Then we can create our onMouseDown function. We want to trigger this when we click on the separator, so let's add our onMouseDown event to the divider.

  const onMouseDown = (e: React.MouseEvent) => {
    setSeparatorXPosition(e.clientX);
    setDragging(true);
  };


  return (
    <div className="splitView">
      <LeftPanel leftWidth={leftWidth} setLeftWidth={setLeftWidth}>
        {left}
      </LeftPanel>
      <div className="divider" onMouseDown={onMouseDown}/>
      <div className="rightPane">{right}</div>
    </div>
  );

Next up is the onMouseMove event. If the flag isn't set we can just return. If the flag is set, then we're going to want to update leftWidth. The value to modify leftWidth by is the mouse's current position minus the stored mouse position. All we need to do is add that value to the leftWidth using the setter. We update the stored mouse position as well, so we can repeat this event for as long as the user is dragging the slider.

  const onMouseMove = (e: React.MouseEvent) => {
    if (dragging && leftWidth && separatorXPosition) {
      const newLeftWidth = leftWidth + e.clientX - separatorXPosition;
      setSeparatorXPosition(e.clientX);
      setLeftWidth(newLeftWidth);
    }
  };

To trigger this function, we need event listeners. We can add those to the document using document.addEventListener. If we create a new useEffect for our SplitView component, then the listener will be added when the component mounts. As an added bonus we can return a function in the useEffect, which is run when the component unmounts, to remove the eventListeners and clean up after ourselves.

  useEffect(() => {
    document.addEventListener('mousemove', onMouseMove);
    return () => {
     document.removeEventListener('mousemove', onMouseMove);
    };
  });

Alright almost there. Finally we just need an onMouseUp function to reset the flag. We'll add eventListeners to the document for onMouseUp events too.

  const onMouseUp = () => {
    setDragging(false);
  };

  useEffect(() => {
    document.addEventListener('mousemove', onMouseMove);
    document.addEventListener('mouseup', onMouseUp);

    return () => {
     document.removeEventListener('mousemove', onMouseMove);
     document.removeEventListener('mouseup', onMouseUp);
    };
  });

And we're done! Congratulations... but at the moment this component just gets a passing grade from me. Let's see what we can do to make it nicer.

Improving the component

Adding maximum and minimum widths

Because otherwise the component is really easy to break by losing the divider out of bounds and not being able to click on it.

To stop the divider from doing this, let's add a minimum width for our panels with a const (if you wanted to get fancy you could pass it in as a prop). Then we just need a bit of extra logic in our onMouseMove function.

const MIN_WIDTH = 50;

  const onMouseMove = (e: React.MouseEvent) => {
    if (dragging && leftWidth && separatorXPosition) {
      const newLeftWidth = leftWidth + e.clientX - separatorXPosition;
      setSeparatorXPosition(e.clientX);

      if (newLeftWidth < MIN_WIDTH) {
        setLeftWidth(MIN_WIDTH);
        return;
      }

      setLeftWidth(newLeftWidth);
    }
  };

So that takes care of the left edge, but what about the right? Well if we had the width of the whole component then we could calculate a maximum width for the left panel and use the logic above. Now how could we access the width of our component…

That's right, using another ref! Let's attach it to the div wrapping all our components.

const splitPaneRef = createRef<HTMLDivElement>();
...
  return (
    <div ref={splitPaneRef} className="splitView">
      <LeftPanel leftWidth={leftWidth} setLeftWidth={setLeftWidth}>
        {left}
      </LeftPanel>
      <div className="divider" onMouseDown={onMouseDown}/>
      <div className="rightPane">{right}</div>
    </div>
  );

Now our onMouseMove logic becomes:

  const onMouseMove = (clientX: number) => {
    if (dragging && leftWidth && separatorXPosition) {
      const newLeftWidth = leftWidth + clientX - separatorXPosition;
      setSeparatorXPosition(clientX);

      if (newLeftWidth < MIN_WIDTH) {
        setLeftWidth(MIN_WIDTH);
        return;
      }

      if (splitPaneRef.current) {
        const splitPaneWidth = splitPaneRef.current.clientWidth;

        if (newLeftWidth > splitPaneWidth - MIN_WIDTH) {
          setLeftWidth(splitPaneWidth - MIN_WIDTH);
          return;
        }
      }

      setLeftWidth(newLeftWidth);
    }
  };

Perfect, no more disappearing dividers.

What is this? A divider for ants?

It's not ideal that our user has to click on an area 2 pixels wide if they want to resize the panels. Maybe we could make it wider?

That looks terrible. How about padding it a little with another div?

  return (
    <div className="container">
      <LeftPanel leftWidth={leftWidth} setLeftWidth={setLeftWidth}>
        {left}
      </LeftPanel>
      <div className="divider-hitbox" onMouseDown={onMouseDown}>
        <div className="divider"/>
      </div>
      <div className="rightPane">{right}</div>
    </div>
  );

.divider-hitbox {
  cursor: col-resize;
  align-self: stretch;
  display: flex;
  align-items: center;
  padding: 0 1rem;
}

Muuuuch better. Changing the cursor also makes it much clearer to the user that this is resizeable!

This is really important for touch screens where 2 pixel precision isn't going to make you many friends.

But none of my touch screen users can use the feature!

That's right! Because we only added mouse events. We're going to need some touch events too!

TouchEvents are pretty similar to mouseEvents so we can reuse most of our code. But since touch screens can support many touches at once, touchEvents store an array of touches. Each touch has its own x and y positions, so we need to tweak our functions a little to account for this.

  const onTouchStart = (e: React.TouchEvent) => {
    setSeparatorXPosition(e.touches[0].clientX);
    setDragging(true);
  };

  const onMove = (clientX: number) => {
    if (dragging && leftWidth && separatorXPosition) {
      const newLeftWidth = leftWidth + clientX - separatorXPosition;
      setSeparatorXPosition(clientX);

      if (newLeftWidth < MIN_WIDTH) {
        setLeftWidth(MIN_WIDTH);
        return;
      }

      if (splitPaneRef.current) {
        const splitPaneWidth = splitPaneRef.current.clientWidth;

        if (newLeftWidth > splitPaneWidth - MIN_WIDTH) {
          setLeftWidth(splitPaneWidth - MIN_WIDTH);
          return;
        }
      }

      setLeftWidth(newLeftWidth);
    }
  };

  const onMouseMove = (e: MouseEvent) => {
    if (dragging) e.preventDefault();
    onMove(e.clientX);
  };

  const onTouchMove = (e: TouchEvent) => {
    onMove(e.touches[0].clientX);
  };

  const onMouseUp = () => {
    setSeparatorXPosition(undefined);
    setDragging(false);
  };

  useEffect(() => {
    document.addEventListener('mousemove', onMouseMove);
    document.addEventListener('touchmove', onTouchMove);
    document.addEventListener('mouseup', onMouseUp);

    return () => {
      document.removeEventListener('mousemove', onMouseMove);
      document.removeEventListener('touchmove', onTouchMove);
      document.removeEventListener('mouseup', onMouseUp);
    };
  });

  return (
    <Container ref={splitPaneRef} className={className}>
      <LeftPanel leftWidth={leftWidth} setLeftWidth={setLeftWidth}>
        {left}
      </LeftPanel>
     <div
        className="divider-hitbox"
        onMouseDown={onMouseDown}
        onTouchStart={onTouchStart}
        onTouchEnd={onMouseUp}
      >
        <div className="divider" />
      </div>
      <div className="rightPane">{right}</div>
  );
};

We can move out the core logic of the onMouseMove function into one shared onMove function that both the onTouchMove and onMouseMove functions can use. Also note that we use the onTouchStart and onTouchEnd events, both attached to the Separator, and reuse the onMouseUp function for onTouchEnd.

One last note is the addition of e.preventDefault if dragging is true in onMouseMove. This stops text being highlighted (or other onDrag mouse events from being triggered) while moving the divider around.

That's great! Can I style it too?

You can! The final touch to our component will be adding a className prop. Drop the component anywhere in your project and style it to your heart's content using your method of choice, whether it be stylesheets, styled-components or something else.

interface SplitViewProps {
  left: React.ReactElement;
  right: React.ReactElement;
  className?: string;
}

export const SplitView: React.FunctionComponent<SplitViewProps> = ({ left, right, className }) => {
...
  return (
    <div ref={splitPaneRef} className={`splitView ${className ?? ""}`}>
>

Happy resizing!

(Cover photo by Fabrizio Verrecchia on Unsplash)

Manage your technical debt roadmap right from your code 🚀

Wed, 21 Oct 2020 00:00:00 GMT

Exec summary:

Technical debt leads to bugs creation: the number of bugs and the number of design flaws (technical debt) are 0.92 correlated in a study from the Software Engineering Institute
Static analysis tools like SonarQube are massively used (100k+ users) to chase quality defects, but they can't replace human intelligence
Bug tracking system are time-consuming and don't help developers while they code
Our tool Tyrion fixes these problems and allows developers to see quality problems right in their IDE

Quality is worth its investment

Martin Fowler in his famous article on quality explains why investing in software quality is cheaper than not investing in it.

Why quality in software is cheaper than no quality

This knowledge has also long been known for lean practitioners: "If you focus on quality, you will deliver a high quality product on time. If you focus on delivery, you will deliver a low quality product late."

Why is that? If you focus on delivering things right the first time you will have less rework to do and then save time in the long run. Teams that don't do that end up spending most of their time fixing bugs.

Low quality leads to too many bugs and missed deadlines

Static analysis tools are mandatory but not enough to monitor quality

Once you know you want to deliver quality on the long run, there are a lot of things to do: identifying best practices, tech trainings, problem solving, quality tooling, setting up all kind of monitoring, choosing quality KPIs ...

If you are serious about keeping the quality of the codebase high, you need to proactively chase, monitor and resolve code quality defects. Even if you have set up all the best linters rules and have a highly trained team, in the long run you will have defects. Indeed, there are a lot of defects that linters can't detect: non homogeneous code and architecture, wrong patterns usage, naming not coherent with business...

A widespread strategy is to monitor quality by static analysis tools like SonarQube or CodeClimate. These tools have some advantages:

✔️You get a quality score without extra work from a developer (once the tool has been set up which can be quick)
✔️Results are objective
✔️They give you a list of piece of code you need to change and which good practices have not been followed

You can have graphs like this one which give you the trend of your technical debt and show you if the situation is getting better or worse.

A SonarQube debt graph from a current project

When talking with tech leads and CTOs, many of them told me that SonarQube has been efficient to help their team keep the quality of their code high.

Static Code Analysis Tools (SCAT) provide objective metrics and insights of the code quality and technical debt. However, these tools require a real integration effort. Such tools without a team adoption and training are of little value. On the opposite, used by trained developers, with a pragmatic understanding of the metrics and warnings, these tools offer a lot of clues on the next refactoring needed to reduce project entropy, on the security breaches still to address and on the potential bugs to review. SCAT warnings and suggestions must be used as starting points of developers' discussions around quality standards. SCAT must not be considered as referees stating good/bad quality points of a project or a team.
— Guillaume Michel, Theodo, VP Engineer

It has also been proven to be effective in some studies like Oswal, Nikhil. "Technical Debt: Identify, Measure and Monitor." (2019)

However, those tools have also limitations:

❌ they create quite a lot of false positive (labeling a piece of code as bad when it is good) and false negative (missing some bad piece of code) requiring times to sort everything.
❌ they can't inspect all aspects of a project. For example some defects like a wrong or non homogeneous directories' organisation are never detected

Academic research while confirming the positive impact of static analysis tools insist that they are not mature enough to only rely on them. For example, to assess if a defect will create a bug, Valentina Lenarduzz warns us in her 2019 study "Are SonarQube Rules Inducing Bugs?": "SonarQube violation severity is not related to the fault-proneness and there-fore, developers should carefully consider the severity as decision factor for refactoring a violation."

This is exactly what Bill Clark explained very well in his article "A TAXONOMY OF TECH DEBT", human can provide much more valuable information about a piece of code with technical debt like the contagious aspect: "Is it likely this bad code practice could spread somewhere else by copy/paste?"

Performant quality monitoring requires human analysis

As we just see, human analysis provides complementary information to static analysis tools:

✔️human can analyse every aspect of the project: architecture, naming, easiness of understanding, maturity of library and frameworks
✔️they can provide much more information about the problems they detect since they understand the context around the codebase
✔️they can provide a prioritization of the problems

The more the developers in your teams are trained about writing quality the more efficient they will be to detect defects. This is another reason why you should invest a lot in training if you want to achieve high quality in your code.

You also need to agree on the best way to write things: what is OK and what is not OK to write in the code. For example, these pieces of code are doing the exact same thing in Symfony.

/**
* @ORM\Column(unique=true)
*/
private $email;


/**
* @ORM\Column(type="string", length=255, unique=true)
*/
private $email;

Both ways have their advantages: the former is shorter to write/read and the latter is more explicit. I prefer the explicit version because I think in this case the tradeoff between the code being super clear and having more words to write and read is worth it. Some other people could say that it is not worth it to add the type and length because it's the default.

Who is right is less important than having your team agreeing in the standard way of doing it. Because once you agreed, people are autonomous to write and check quality on their own regarding this standard making little by little the whole code base coherent.

The main drawback of human analysis is that they require a lot of effort to get and update the information about the overall quality.

Comparison between static analysis and human analysis for software quality

Existing tools are time-consuming and don't help devs when they are coding

At Theodo we experienced several systems to track self admitted technical debt (SATD, as software quality researchers calls it).

On small or short projects, we use a special column in Trello/Jira/Monday/Google doc to track tasks we need to do to fix quality problems. This works OK as long as the team fixes everything quickly, so the column doesn't get too long and messy.
When the project grows we quickly need a real tech roadmap on a separate board.

Example of debt management using Trello

We can also use a specialized bug tracking tool like Bugzilla or Bugherd. It works well, but it has two drawbacks:

❌ it is time-consuming to update it when an item has been fixed or is no longer valid
❌ developers don't see the defect in their IDE while coding and thus could copy/paste it creating the contagious aspect of the debt explained by Bill Clark.

Introducing Tyrion: a tool to help teams track software quality through human analysis

I initiated Tyrion to remove those main drawbacks and be able to:

✔️Document the debt while coding directly in the IDE
✔️Painlessly update the debt tracking system when something has been fixed
✔️Quickly get an overview and the trend of the debt

Writing comments in the code to map the debt

Instead of registering all technical defects in a tool, the main idea of Tyrion is to add them directly in the code base as comments. Tyrion will parse these comments to generate the debt graphs or CSV files so you can analysis the debt.

The syntax of the comment is straightforward: the only thing you need to add to a classic comment is a type to allow Tyrion to group them:

// TODO DEBT_TYPE "Author: comment"

I like the following example from a current project. Albéric and Emyly are working on improving the accessibility of the webapp. Nothing urgent from the business and nothing requires immediate change from a technical point of view. But still, they know they can do this piece of code better, so they mark it as debt to improve it a bit later.

// TODO accessibility "Alberic: Add keyboard behaviours and focusability"
export const Select: React.FC<Props> = ({

Even if you don't use Tyrion, adding debt comments in the code is valuable by it-self. It trains developers to evaluate the quality of the code while they are coding. It also partially protects you again the contagious effect of the debt because if you have correctly marked with comments the debt in the code, developers will think twice before copy/pasting a piece of code that is marked as a debt.

If you have a large code base, it can take a while to document the whole technical debt. You can do it gradually during few weeks. While working on some part of the app each member of your team will read different files and find something to mark as debt. By doing that you will create little by little the debt map of your project. Of course, it's better to fix a problem when you see it but for various reason you don't always have the time to do it.

If your project is of reasonable size, you can also organize a big debt mapping session one afternoon.

Using the CLI to generate the first graph

Once you have commented at least a part of the debt you have in your project, you can start using the Tyrion CLI.

First you need to download it with: npm i -g tyrionl or yarn global add tyrionl

Then just run tyrion in the root directory of your project. It will scan your files to look for TODO, FIXME or @debt strings in the comments and then assume it is a debt comment.

This first mode is useful to get information of the current debt status of your project. It can help you respond to question such as "How much debt do we have right now?" or "What kind of debt do we have most?"

If I run Tyrion in the frontend part of our webapp, I got the following graph where we can find the acessibility type of debt from the above debt comment among other types we use on the project.

An example of the debt graph showing the current distribution of types of debt

You can customize the weight of each type on the global debt score

All kind of debt doesn't have the same impact. A security issue can be more critic than an over complex function. So you may want to weight differently each kind of debt. This can be done by creating a .tyrion-config.json file in the root of your project. You can then assign different value to each type of debt.

Here is the default configuration:

{
  "pricer": {
    "bug": 100,
    "architecture": 100,
    "bugRisk": 5,
    "security": 100,
    "securityRisk": 10,
    "quality": 5,
    "test": 5,
    "doc": 3,
    "ci": 30,
    "deploy": 10,
    "devEnv": 10,
    "outdated": 5
  },
  "standard": 100,
  "ignorePath": [
    "node_modules",
    "README.md"
  ]
}

You can find more information on how to use Tyrion on the README

The trend of the debt may be the most important information

After a while of commenting debt in your code, you will be able to use the second mode that will analyse your Git history and will create a graph of the evolution of your debt. The command to get it is tyrion -e 200. You can replace 200 with the number of days you want to scan backward.

An example of a debt report showing the evolution of the debt

This kind of report will help you and your team respond to the question "Are we creating debt or increasing quality?" It is important to know because if the trend of your debt is increasing, you should invest more in quality. Remember you won't go faster by overlooking quality 😉.

Sharing it with the product/business team

If you are reading this, you are probably a developer, and you probably want to spend more time on quality. You may not be able to do so because someone in the business side of your project prefer that you focus on a short term deadline.

So you need to negotiate with this person. This tool was also build to help you to do so by providing graph and data to support your case.

If you need a csv file to allow business people have a closer look at the data you can generate it with the option `--csv`.

Sharing quality data with product team is a good way to onboard them on the topic

Conclusion

Keeping a high level of code quality on a project for years is something very difficult. One positive side effect of using Tyrion on a project is that developers can grow their quality awareness while coding. If all developers of your team ask them-selves "Is this piece of code I'm writing/reading the best we can do?" for each feature, you will be on a good track to keep the quality high on the long run.

You have a question? Feel free to either contact me:

📧 by email
🐦 on twitter
🐙 by leaving an issue or a pull request on the Tyrion repository
💬 by leaving a comment below

I will be happy to discuss about Tyrion and quality management with you.

Write tests for humans

Mon, 07 Sep 2020 00:00:00 GMT

Let me tell you the story of the day I almost quit testing my code altogether. For the tenth times this week, the CI failed with this cryptic error message "Received value does not match stored snapshot" and a diff that looked something like:

<div
  - className="myClassName" >
  + className="myNewClassName" >
  + <div>
    Hello world
  + </div>
</div>

It felt pointless to fix this test : why does it matter if the class-name is different or I added a div tag? It does not keep me from introducing regressions as nobody in the team reads the snapshots updates.

I was so frustrated, I almost broke the computer, slammed the door to quit my job. But instead, I decided to channel this energy into something more positive. I started thinking about how the project tests got into such a state.

How come everybody agrees that we need to test our code base yet so many developers hate it? Was I the only one being frustrated? (No I was not)

When people say they use TDD, I don’t believe them.
— emily freeman (@editingemily) August 26, 2020

Why is nobody doing something about this?

The Vicious Circle Theory or How you stopped writing test for humans

Here is my theory to make sense of this kind of situation. I have called it "The vicious circle of testing" ™:

A Developer changes the code for a feature. A test fails, but it is not clear why.
The Developer fixes the test. They do not know why the test was here in the first place. They grow used to the idea that it is not worth their time investment to fix it.
Next time, this developer writes code, they spend less time and effort writing tests.
And back to the top...

If this resonates with you, you've been writing tests for the CI to work but not for your teammates to understand your intent. Tests are a collaboration tool, they are meant to be read when they fail by you of your teammates. You need to make sure you can understand them in a month when you have forgotten all about what you were supposed to do.

Tests are meant to be read by humans, not your CI.

How do you write tests for humans, you ask? Here are my 3 tips for you :

Why: You need to know why you are testing. Because testing is a universal truth in software engineering, it is easy to forget why you need to do it on your project. Find what motivates you: it will guide you to write good tests that you want other humans to read.
What: You need to focus. I think it is better to do one thing well rather than try to do it all and do it mediocrely. Choose what you want to test, do not try to test it all.
How: Choose the best tools adapted to your goal. They must help you write quality tests that you can read and maintain

#1: Why do you write tests?

Writing great tests for humans takes time, you need to be invested in what you are doing. You cannot be if you do not know exactly why you need to write great tests.

Do not write tests because you have to. Make sure you and your team know why you write tests.

Automatic testing is often a given when setting up quality tools for your project. Unlike your linter, tests do not fail every day, they are a longer-term investment to code quality. It is easy to lose sight of why you are doing it.

Before starting to set up your test environment you should clarify why you are testing.

What are you hoping to gain in the long term? Better documentation? Fewer regressions bugs? Developer experience tool for TDD?

Make sure you and your team know why you are doing it. Because it is so easy to fall into the vicious circle of bad testing, if you all know why you are driven to write tests, you can catch early when you are deviating from your goal. Making your goal clear will give your teammates ownership over your test and they will help you push for improvements when needed.

#2: Choose what parts need to be tested first

If we chose to use snapshots on my project, it was to go faster. We wanted to test all of the codebase without investing much time. By aiming to keep a 100% coverage, we started writing tests that needed to be maintained every time we added features. They could not help us catch regressions because they changed too often. No matter how small the investment was, it was not worth it.

Do not aim for a 100% code-coverage of generic tests, aim for 100% of readable tests

You do not need to test everything. Your set up may not be adapted to every aspect of your code.

I always write unit tests for very logical functions :

They are hard to read and easy to break. A test is a safeguard to make sure another developer will not break it.
It is cheap to set up and to write.
It is fast so it does not slow down the ci.

But for most part of the code, it is a case-by-case study.

Should I test my frontend app UX? If I am writing a complex single-page app, UX is very likely to evolve. Testing it may be my priority so I can prevent regressions that have a big impact on the user's appreciation of my app. I want to write integration or end-to-end tests for this.

Should I test your frontend app UI? If I am writing a shared-component library, I am focusing on UI and have a few logical components. My first mission is to provide my users with reliable and consistent UI. Testing UX logic is a bonus, it is ok to make choices and not to focus on this at first.

#3 Choose your tools right

There are a lot of great testing tools available. Now that you know why you want to write tests and what parts of code you want to focus on, you can find the tools that fit your needs like a glove

On another project, I chose to test my frontend React app with react-testing-library. I wanted a tool that :

would test our React app components logic and document what they did
would help new developers feel confident when starting on the project
would help us focus on these goals even if the teams changed

That is why I chose react-testing-library to write integrations tests on it. It turned out to be a great developer experience. I also had a personal goal of trying this new tool that had a lot of great reviews.

Find a tool that makes writing test fun, or that makes you want to do test-driven-development, you will only write tests for humans this way.

You are ready!

Now you know: if you ever find yourself thinking testing is useless or painful to maintain, you are probably in a very vicious circle of testing. If you want to escape this, here are the steps to follow:

Remember tests are a documentation and collaboration tool, they should be made to be maintained by teams and read by humans
Make sure your team knows why you are writing tests. This will motivate you to write good tests that you can be proud of.
Do not try to do it all. Coverage is just a number, it is not a token of code quality. Focus on testing some parts well rather than all of it in a poor way.
Choose the tool that fits you and that makes the developer experience fun

Software-based CPU power consumption using PowerApi

Thu, 03 Sep 2020 00:00:00 GMT

The ICT sectors carbon footprint represents now around 4% of the global CO2 emissions. To minimize that, one of our goals at Theodo is to design sustainable web applications with minimal carbon footprint. And in order to know how performant we are at this, we need to be able to evaluate the power consumption of our applications and more specifically the power consumption of the VMs we use in the cloud. Unfortunately, at the moment no cloud provider is able to provide such a data. So we had to look for a solution.

Our research led us to PowerApi, an open-source project developed by the Spirals research group from University of Lille 1 and Inria in France. It allows people to estimate the CPU’s power consumption induced by an application without any external device like a wattmeter. As we can't put wattmeters to measure the VMs consumption in the cloud, this looks like a good option.

The magic formula

From the documentation, we can read that PowerApi collects with a sensor raw data from the hardware of the server. Then PowerApi applies a “formula” and voilà: we have the power consumption of the monitored software stored in a database.

After reading this I asked myself two questions : what is this “raw data” and how did they build this magic formula ? Fortunately, we can all have access to their research papers where they explain everything. So let’s dig a bit deeper.

A Power Model is used to approximate the power consumption

The formula is what is called a Power Model in the academic field. A power model is a model that approximates the power consumption of a hardware based on different factors.

I knew that each CPU has a specific TDP on their specification page: “Thermal Design Power (TDP) represents the average power, in watts, the processor dissipates when operating at Base Frequency with all cores active under an Intel-defined, high-complexity workload”

So when I pictured the formula, I thought it was going to be quite simple and something like:

Consumption = Idle power consumption + %CPU load * (TDP - Idle power consumption)

Reading their research paper, it turns out it’s more complicated: “the CPU load does not accurately reflect the diversity of the CPU activities. In particular, to faithfully capture the power model of a CPU, the types of tasks that are executed by the CPU have to be clearly identified.”

Power consumption is related to the CPU activity which is not exactly the CPU load

The types of tasks in a CPU are also known as events. Each processor type has hundreds of different events with potentially a different power consumption. To have an acute idea of what the CPU is doing and thus knowing the related power consumption, the PowerApi teams decided to rely on those events and count them.

To do that they rely on specific counters of the CPU: the HPCs (hardware performance counters) : “We therefore decide to base our power models on hardware performance counters (HPCs) to collect raw, yet accurate, metrics reflecting the types of operations that are truly executed by the CPU.”

Learning phase: creating the Power Model from the regression analysis of real power consumption

The goal of this phase is to approximate the amount of power consumption per HPC events during a workload. It’s made possible by measuring both real Power consumption of the server and counting HPC events in real time.

Step 1: select an unbiased set of workloads

To ensure that the data is not biased to one kind of CPU activity, the team used an academic set of different workloads called PARSEC.

“To explore the diversity of activities of a CPU, we consider a set of representative benchmark applications covering the features provided by a CPU. In particular, to promote the reproducibility of our results, we favor freely available and widely used benchmark suites, such as PARSEC.”

Step 2: get the correlation between real power consumption and HPC events

The team measured the real power consumption using a bluetooth power meter : the Power Spy 2.

Real time in the context of this experiment is that every 250 ms:

The Power Spy 2 send the power consumption of the last 250 ms to a collector
The number of HPC events of the last 250 ms are sent to a collector

By doing so we obtain a lot of data linking power consumption and HPC events.

Step 3: select the HPC events that are the most correlated to power consumption

The two final steps are a mathematical ones. Step 3 is about removing the HPC events that are weakly correlated with power consumption to simplify the step 4.

Step 4: Create the Power Model using ridge regression

At this step, the team uses an advanced mathematical method of regression (ridge regression), to get the 2 or 3 HPC events that will be able to predict the power consumption of the software on the machine. This is what we call the Power Model

Here is a diagram of the whole phase:

The final Power Model includes memory, disks, motherboard and fans

As the Power Spy 2 measures the power consumption of the whole server, all other hardware components (memory, disks, motherboard and fans, ..) are integrated into the power model. It works well because the CPU is the most energy consumming component.

The final power model is P = Pidle + Pactive, where Pidle corresponds to the power consumption of the CPU when no program is running and Pactive to the additional power consumption from a workload.

For example, one power model for the Intel Xeon W3520 is

With e1 and e2 two HPC events.

To get a real time power consumption of this server PowerApi has then just to count those events.

Real life application

As Cyrielle has explained in this blog post, we were able to measure the power consumption on our local machine. But we couldn't use it to calculate power consumption on AWS servers which was our original goal. Indeed the research team can't do the learning phase on the AWS servers (yet?) as they don't have access to it. So the PowerAPI software can't get the data needed for its Power Model.

Thanks to this project and the things we read to try it, we learnt a lot:

The CPU is the component in a server that consumes the most (~100 Watts). The memory is around 10 Watts for 32GB and a SSD consummes around 2 Watts.
CPUs, when idle still consume around 1/3 of the peak consumption.
The next generation of Intel's CPU may see a 91% reduction in power consumption
ARM processors have a very low energy consumption compared to x86 processor(~2 Watts vs ~100 Watts)
AWS provides arm-based EC2 instances if you look for efficient way of reducing the carbon impact of your web application.

The next idea that looks promising to have a rough idea of the power consumption of our application is the "Cloud Jewels" approach by Etsy adapted for AWS.

Prevent AWS from Reading Your Step Functions Data

Mon, 31 Aug 2020 00:00:00 GMT

AWS Step Functions is the perfect tool to handle long-running, complex or business-critical workflows such as payment or subscription flows. However, a naive implementation could put sensitive data at risk.

What is AWS Step Functions?

AWS Step Functions is an Amazon cloud service designed to create state machines to orchestrate multiple Lambda functions, branching logic and external services in one place. It presents most of the advantages of the serverless services: the scaling is immediate and the pricing is based only on the computing time and the number of step transitions.

The state machines can be defined directly in the AWS Console with a JSON configuration and you directly have a graph representation of your workflow.

AWS Step Functions state diagram

At Theodo, we use Serverless as our framework with the Step Functions plugin to write our AWS Lambda functions, define our resources (DynamoDB and Queues) and setup AWS Step Functions and the state machines directly in our codebase.

Example: an identity verification workflow with Step Functions

The previous picture shows an identification verification workflow. It is a piece of a bigger validation flow for banking applications.

What it does is request an identification URL to a third-party system, send this link by text message to the applicant and wait for them to complete the identity verification. When it is done, the third-party system calls a webhook, which restarts the state machine and either triggers a retry, sends a rejection email, or continues the global validation process.

The identification flow

To perform all of those actions, sensitive information will flow through the inputs and outputs of the state machine steps:

👤 Personal information from the applicant which are sent to the third-party service to check his/her identity
📱 The phone number of the applicant to send the SMS
✉️ The email address of the applicant to send the mail
✔️ The result of the identity verification

Problem: we are exposing sensitive data

With this implementation, we see that sensitive data flow through the inputs and outputs of your state machine directly.

state-machine.yaml
----------

SendIdentificationRequest:
  Type: Task
  Resource: arn:aws:states:::lambda:invoke
  Parameter:
    FunctionName:
      Fn::ImportValue: ${self:provider.stage}-sendSMS-Function-arn
    Payload:
      creditApplicationId.$: $.creditApplicationId
      creditApplicant.$: $.creditApplicant
      phoneNumber.$: $.phoneNumber
      message.$: $.smsBody
  ResultPath: $.identificationSMS
  Next: WaitIdentification

An exemple of sensitive data flowing through our inputs and outputs

The problem is: nowadays, most DevOps engineer, developers and even some product managers with access to AWS and can see that information. Furthermore, AWS Lambda dumps are stored in an S3 bucket which is not encrypted with the default configuration. Any malicious access to your AWS stack could leak sensitive information far more easily than by corrupting your database for example. The more you spread sensitive information between multiple services, the more you put yourself at risk. Finally, as it is stated in AWS documentation:

Any data that you enter into Step Functions or other services might get picked up for inclusion in diagnostic logs.

Depending on your data classification or threat model, this could be totally unaceptable. With that in mind, you may completely dump AWS Step Functions, or you may try to find a solution because the tool is still awesome and perfectly suits your needs!

Our solution

To prevent leaking sensitive information, we need to clean every step function input and output. Here are three ways of doing it:

Configure Step Functions to log in CloudWatch Logs instead and set a low expiry date. This is the easier solution but it still exposes logs for a short duration, and you lose any chance of retrieving old execution informations later.
Keep the data in the input and output but encrypt and decrypt it between each step. This one is a bit more complex but you only need to use an encryption key in your lambdas. However, you are still exposing encrypted data, which can be insufficient for some critical sectors, such as health or defense.
Never transmit sensitive data between steps and retrieve it directly during their execution. This solution requires more services to work and thus more things that could fail. However, it is the more secured and allow permanent recovery of logs.

Second and third solutions also requires break-glass processes to allow developpers to legitimately access log data, for debugging or support matters.

Considering everything, we prefered to implement the last solution because we wanted to keep our sensitive data in a single safe spot, where we put a lot of attention to the security. Also, we were already using secured AWS DynamoDB encrypted with external KMS keys for the waiting tokens of our lambda functions. Thus, we decided to create a new table to store sensitive data between steps.

We configured it like below.

resources.yaml
----------

StepSentiveInputTable:
  Type: AWS::DynamoDB::Table
  DeletetionPolicy: Retain
  Properties:
    TableName: step-sensitive-input-table
    AttributeDefinitions:
      - AttributeName: sensitiveInputId
        AttributeType: S
    KeySchema:
      - AttributeName: sensitiveInputId
        KeyType: HASH
    BillingMode: PAY_PER_REQUEST
    PointInTimeRecoverySpecification:
      PointINTimeRecoveryEnabled: true
    SSESpecification:
      SSEEnabled: true
      SSEType: KMS
      KeyId: hsm-key-id

DynamoDB table definition in Serverless

We were already using DynamoDB from our lambdas to save our waiting task tokens and we chose to add another table to save sensitive data between steps. Furthermore, on AWS, you can choose to save the encryption keys of your DynamoDB tables outside of AWS, which was a big plus for us.

To avoid repeating ourselves, we coded a wrapper that we use on all our lambdas. It does the following things:

After the return of the lambda, the wrapper:

Looks for sensitive data keys in the output, remove them and build an object with that information
Save the sensitive data objects in a dynamo entry
Add the id of the new dynamo entry under a “sensitiveData” key in the output

Before the execution of the main lambda function, the wrapper:

Looks for “sensitiveData” keys in the input
Retrieve the content at the given IDs
Replaces the "sensitiveData" keys with the resulting object in the input

// encryption-wrapper.ts

export default (lambdaHandler: LambdaHandler): LambdaHandler => {
  const decryptHandler = async (event: any) => {
    const decryptedInformation = await getSensitiveData(event.sensitiveInputId);
    return lambdaHandler({ ...event, ...decryptedInformation });
  };

  return async (event: any) => {
    const clearOutput = await decryptHandler(event);
    const sensitiveInputId = await storeSensitiveData(clearOutput);

    for (const sensitiveKey of sensitiveInputKeys) {
      delete clearOutput[sensitiveKey];
    }

    return { sensitiveInputId, ...clearOutput };
  };
};

Our custom encryption wrapper

Last of all, we needed to encrypt the first input at the start of our state machine from the API to finally secure our users information all the way!

Takeways

Take care about what information you have in your lambdas and state machine events on AWS
Always think twice about the data you put in your logs
Spread your secrets and security implementation in as few services as possible
Use a wrapper if you want to repeat behavior in multiple lambda functions.

Thank you very much for reading my article. Feel free to comment, ask for implementation details, and feel free to share the problems that you encountered with AWS Step Functions.

Typescript: use the nullish coalescing operator to prevent bugs

Tue, 04 Aug 2020 00:00:00 GMT

My goal as a CTO is to improve quality. The score of this game is the number of bugs we find each week. Today I share with you a typical bug that more than one person got caught by.

Let's say you want to initialize the audio volume of your react application with the value previously saved in the localStorage or defaulting it to 0.5 if nothing was saved. You could write something like:

Bad example

function initializeAudio() {
  let volume = localStorage.volume || 0.5;

  // ...
}

The problem is that if the user has saved their volume to 0 it will be interpreted as falsy and your code will use the default value instead of the right value. Some devs prefer the simplicity of || than putting an explicit if clause. And more than once, they were right because 0 was not a plausible value. But as a standard it is too dangerous. For example someone else could see the code and think the || is a good coding stantard in all situation which will eventually create a bug.

Typescript 3.7 comes with the best option: you can now use the nullish coalescing operator to prevent that wrong behavior and safely write something like:

Good example

function initializeAudio() {
  let volume = localStorage.volume ?? 0.5;

  // ...
}

That will be compiled in:

function initializeAudio() {
    var _a;
    var volume = (_a = localStorage.volume) !== null && _a !== void 0 ? _a : 0.5;
    // ...
}

To make sure people from your team use it, you can use the prefer-nullish-coalescing ESLint rule. You can also use it with javascript and this babel plugin. If you want to go deeper in the understanding of this operator you can go here.

I hope this article will help your team prevent this kind of bugs. For more actionable code quality tips, find me on Twitter.

Implement contract testing with Pact to get rid of bugs due to front-back and back-back communication

Sun, 05 Jul 2020 00:00:00 GMT

Image from the Pact documentation

Exec summary

Modern software development organizations scale their development efforts by spreading the development of a system across different teams. Such projects can have team work issues because interconnected applications that change often, create a lot of errors breaking other teams' application. A typical problem due to a large organization is when the backend team deploys a change that is not compatible with the mobile app and breaks it.

Contract testing is a way to make sure that several applications developed by collaborating teams can understand each other. With no need to take extra manual care when modifying code, the teams avoid human errors, get rid of bugs, and on the way, develop faster.

With contract testing, a team can easily create a test checking that the application they are developing receives viable responses from the provider application that another team is developing. On their side, the provider application team only has to run a single automatically generated test. This allows both teams to know if something is broken before they deploy their applications: they can talk quickly about this precise problem, with little rework cost and no bug for the users.

Pact is the main framework to implement contract tests, and it works with many languages.

For Whom?

In order to avoid creating bugs coming from miscommunication between two parts of your applications you can set up different tactics, contract testing is one of them. Here are some alternatives with an evaluation of how they perform. You can find some comments on these alternatives in the appendix at the end of the article.

Alternative	Maintenance cost	Setup cost	New interaction cost	Learning curve	Efficient
End-to-end testing	$$$	$$$	$$	$$	++
Shared types	$	$$	$	$	++
Versioned API without breaking change	$$$	$$	$	$$	+++
Massive documentation and conception	$$$	$	$$$	$$$	+++
Contract testing	$	$$	$	$$$	+++

All these tactics can be used together where it is possible.

In fact contract testing needs some conception to create the contract or update it. But contract testing allows to have an accurate documentation when you want to do this conception. Once you discover a breaking change you still have to take a decision on how to handle it and versioned API are a good way to do it but the contracts will show you what you do not need to maintain because no one uses it anymore.

Conclusion: if you develop a distributed system communicating via HTTP APIs with heterogeneous technologies you should find someone to help your team set up contract testing.

Benefits

Benefit #1: Defects are stopped before wasting others time

Usually to achieve testing of several services interacting, we use some kind of end-to-end testing. This means you can test only after deploying your service to put it on the same environment as other services (e.g. staging, preprod, CI). What is called end-to-end testing here could be called broad stack tests, it matches the definition found here. If there are several teams, it becomes difficult to have someone responsible for end-to-end tests.

With contract testing you can test directly in your continuous integration (CI) workflow which means no one else than the developer invested time in checking the work done, this allows problems to be caught earlier. (cf. process just below)

doing > manual local test (few minutes) > push (few seconds) > CI (few minutes) > code review (some minutes) > merge (few seconds) > deploy (few minutes) > manual test on staging > validation by the PO

Benefit #2: Developers know their mistakes and correct them quickly

As you can see on the process above having the result in CI instead of after deployment creates a faster feedback loop for the developer and he can correct it before the merge which means corrections are done faster also.

Let me introduce the pyramid of tests briefly, there are several kinds of tests with different drawbacks, the idea is that even if useful, the more drawback you have, the less tests of this kind you do.

Image from the Martin Fowler article Test Pyramid

The main advantage is that contract tests are lower on the pyramid of tests (between unit for the cost and service for the speed) than end-to-end tests (UI). It means they are faster and cheaper, so we can test more cases which gives quickly more confidence for less money.

Benefit #3: Contract tests are easy to create and maintain

With the pyramid we saw that contract testing is cheap compare to the end-to-end alternative. Our experience is that once set up the cost is close to unit tests (only slower) which are widely used on most projects.

Each test, tests exactly one case, this means they do not change often (compared to end-to-end tests where any change can break the test) and that a change is easy to do in the test because the complexity is low (like a unit test).

Benefit #4: Contract tests give auto-documented API usage

Thanks to Pact that registers all your contract tests, you know precisely what are the interactions tested in your project. This is up to date even before going to production and you can easily visualize it from Pact.

Services interactions:

Image from the Pact documentation

Details of an interaction:

Image from the Pact documentation

Trade-offs

Trade-off #1: It is useful only when several teams work together

Tests are used to perform check that are hard to remember and take a lot of time to perform manually. For contract tests this means they are less useful when you only have one or two teams as you usually know exactly what the other team uses.

They remain relevant but the cost for an inexperienced team on how to use them could create more problems than it solves at first. For an experienced team the cost is not an issue making them a practical solution even to test the interactions between one frontend and its API developed by the same team.

Trade-off #2: The setup is quite complex at first

To make contract tests work you need several days of work.

All teams need to understand the core concepts and how to write good tests with the good level of testing.
You need to add a pact broker in you environment (meaning you have to deal with security issues and keeping it running) this is mitigated by the fact the provided docker container works fine.
You need to make sure your providers applications run in your CI. By default there is no need to have a running instance in your CI but this is quite easy to do with docker.
You need to find a way to handle state in your provider apps to simulate all kinds of responses for instance you need to simulate your own dependencies to make the test reliable. Depending on the complexity of your dependencies (number of dependencies and number of possible outputs) this can be more or less difficult.

My Opinion

You can benefit of contract testing in many situations. Once you have overcome the initial cost to set up and understand how you should work with it for a big project it can even be a cost effective solution for smaller projects in your organization. So join the Pact users' community to improve your test stack and help your teams to ship faster. You are not the first, many companies like Atlassian use this tool already as they explain it in this video.

Go Further

To know if you can get benefit from Pact you should read this page of the documentation, in case of doubt read this. Then to learn how to use it you should begin here and here. To know where these tests should be used you can read the excellent article The Practical Test Pyramid which describes most kinds of test you can use in a project.

Appendix

If you want more details, here are some elements I used to create the evaluation table of the alternatives in the "For Whom?" part:

End-to-end testing

Pros:

When the tests is green you are sure everything works together since the tests checks everything at once

Cons:

The cost of maintaining an environment to test everything.
Test tend to be very flaky due to external causes.
You need to run all tests (and they are slow) before the real deployment of any part, which creates a congestion in your teams workflow.
The coverage of error cases costs even more because you need to set up your system to fail in expected ways and because each case is quite long to execute you tend to have few.
When you discover a real bug, you still need to work to find the root cause (slow to fix)

Use: When you want to check your most critic happy path and you have a way to have enough control on your system in an environment before prod (set up mocks for external services, set up well known data...)

Shared types

Pros:

Easy to set up

Cons:

You do not have the vision of what versions work together, this force to deploy the different pieces together which is a smell for down time.

Use: When you have a small enough system that you can deploy all at once and a technical way to share code between consumers and provider.

Versioned API without breaking change

Pros:

The problem cannot exist

Cons:

Maintenance cost is high, you either have to live with the burden of all your previous choices or find a way to know what is not used anymore (one way is a deprecation plan of older versions).
Risk of human error and still create a bug

Use: For public APIs this is a good solution as there is no way to contact each consumer directly.

Massive documentation and conception

this kind of massive

Pros:

Reduces the amount of bugs of any type.

Cons:

Forces you to make the perfect conception, keep track of every decision, to have comprehensive checklists and to train your whole team to follow all items.
Your process needs to be at the highest level, like the NASA for space shuttles.

Use: When lives depends on your software working perfectly

Contract testing

Pros:

Catch bugs between teams before deployment
Easy to maintain
Teams can deploy at their own pace
Documents expectations of all consumers and ensure they are met with no human intervention
Keeps track of what versions are compatible

Cons:

You need to carefully set up your workflow to integrate these tests.
You need to create tools to handle complex cases.
There is yet another tool to teach to your team.
You still need other kinds of tests to check the business rules of each application work as expected

Use: When you want to quickly develop a system with more than two interconnected applications (if you do not control the update of some applications like for mobile, each used versions count). And each application is developed by independent teams which communicate frequently. It can be better to have someone to train your team to reduce the setup cost.

Serverless Is More Than a Trend, Best Companies Adopt It

Mon, 29 Jun 2020 00:00:00 GMT

A recent article by the WSJ highlighting the benefits of serverless for organizations points at a building awareness by thought leaders about the impact this new trend can have for companies. It resonates with our own experience, celebrating successful serverless implementations with our clients in the last couple of years.

Yet, when pitching prospects about the value they can unlock through this technology, we often find that serverless is not very well known among decision-makers.

If the value is clear, why is there a lack of awareness?

The Term Serverless is Confusing

Ben Ellerby defended the serverless concept in an article, noting that it “is not a good term, yet it is used to describe a powerful and often misunderstood concept.” So, here is his definition:

When we first introduce people to serverless, there is this aha! moment when they understand it doesn’t mean there are no servers -- it’s just that they’re further outsourced to cloud providers.

This graphic positions serverless in the cloud landscape:

In a nutshell, using serverless gives you the ability to develop your own applications and microservices, to integrate off-the-shelf services like payment or user authentication, and to process events generated by these services or other SaaS. Meanwhile, the size of the engineering team decreases, as the full responsibility of managing the infrastructure is delegated to a cloud provider.

Serverless, Right for Large Companies as Well as Start-Ups

Gartner already mentioned serverless computing as one of the top 10 trends in 2019. Allied Market Research expects the global serverless architecture market to reach almost $22 billion in 2025, from $3 billion in 2017. Led by IT & Telecom, they expect the revenue to increase in all sectors, with the highest growth in the Media & Entertainment industry.

Large companies drove the market at its beginnings, with use cases such as real-time web applications and data processing; but SMEs will exhibit the highest growth rate between 2018 and 2025, according to the same study.

In his article Serverless is for Everyone, Sheen Brisals, Senior Engineering Manager at The LEGO Group, explains that “for a majority of start-ups, the tech stack is something they are born with”, therefore “change is easier when compared with a big enterprise”. But “starting small with serverless is perfectly applicable to a start-up as well as to a grown-up.”

As cloud providers invest in improving their solutions, more developers adopt serverless for developing microservices and functions, expanding the potential use cases.

Download the free presentation here

It’s Not Just About Saving Costs. Serverless Computing Solves Business Problems and Delivers Better Customer Experiences.

Reduced operational and development costs are the most important benefits sought by organizations going serverless, together with the automatic scaling of servers according to demand. On the product side, the increase in developer productivity and reduced engineering lead-time has a positive impact on the speed of delivery, thus on the time-to-market.

To overcome the challenges of dealing with legacy applications built during their 100+-year existence, companies like Liberty Mutual embraced serverless computing to solve real customers’ problems. Among the solutions that the IT team have built, Gillian Armstrong (Solutions Architect at Liberty Information Technologies) and Mark McCann (Architect) mention:

A virtual agent for their call center to answer the most common questions.
A virtual assistant to help the clients manage their policy documents.

According to Armstrong, with serverless:

The cost is so low, because it's all managed services, and they're able to bring it out, trial it with users for a little bit, make sure the whole system works, and then just scale seamlessly up.

At Theodo, we're also obsessed with business outcomes, and we train our engineers to focus on delivering value for our clients' end-users, in line with our Lean culture. When we start a new project, we recommend our clients use serverless components when we're confident it will add a competitive edge to the solution.

Check in this video to see how our ground transportation client used serverless for managing omnichannel, real-time notifications to passengers:

Serverless Computing Is as Sophisticated as It Is Powerful: the Main Roadblock is Training

According to the O’Reilly Serverless Survey 2019, with a panel of 1500 respondents from a wide range of locations, companies, and industries, some of the biggest challenges are linked to the education and hiring of staff.

In the organizations that have not (yet) adopted serverless, “fear of the unknown” comes in second after security concerns.

Some components proposed by Cloud providers, like DynamoDB from AWS, have huge potential, but using them correctly is still a skill that requires mastery.

To tackle these concerns, Theodo developed EventBridge Storming workshops, adapted from the Event Storming workshop technique by Alberto Brandolini. During these sessions, we work with business stakeholders to define a common language around application fields (e.g. order, payment, and shipping) that we can use to communicate with the team. We then use this vocabulary to design an architecture based on events grouped by domain and well-defined system boundaries.

Cloud providers and the user community also recognize the challenges faced in upskilling teams to handle serverless. The good news is that they are doing an outstanding job at improving the tools, developing best practices around security concerns, and sharing knowledge on novel ways to solve issues. Theodo is proud to have contributed to multiple frameworks and libraries in the space, an achievement recognized with a partnership with AWS and the Serverless Framework.

Conclusion

Reductions in Total Cost of Ownership and Time-to-Market are two compelling reasons to consider serverless for IT leaders. Besides, the granular nature of FaaS and managed services allow companies of all sizes to progressively build new applications and replace pieces of legacy architecture without requiring an engineering big-bang.

Chances are that you're already using serverless. If your technical stack includes a cloud provider such as AWS, Azure, or Google Cloud, your engineers have probably used their FaaS offerings for data processing or queue services for notifications. You're not starting from scratch!

This means that serverless is one of the easiest technologies to adopt. Even if your main infrastructure is still bare metal, you can experiment and iterate with serverless on a limited scope quickly and seamlessly. It will give you a better idea of the benefits and challenges of serverless for your company, right now.

Need help? Contact me at alexdb@theodo.com

Cover photo by Danielle MacInnes on Unsplash

Build an Animated Accordion List in React Native

Sat, 27 Jun 2020 00:00:00 GMT

Accordion lists (or ExpandableListViews) are now a material design staple and show up in quite a few apps. They are a great way to dynamically display information to a user.

When it comes to implementing one in your own app, sure, there are libraries that you can use. But implementing one yourself gives you greater control over styling, a smaller app bundle size, and it’s so easy it might even be faster! It’s also a great opportunity to sharpen your skills using React Native Hooks and the Animation API.

This is what we’ll be creating:

Just show me the code!

Alright, alright! Here’s a snack. Consider sticking around though - you might learn something.

Let’s get started

First off we need a functional component. This will take two arguments or props:

A string. The title in the top bar of the list item.
A ReactNode. The component in the expandable section.

For the second of these, React allows us to access child components via this.props.children in class components, or we can specify children as an argument to our functional component. Magic!

So our functional component has the form:

const AccordionListItem = ({ title, children }) => {
 ...
};
export default AccordionListItem;

And we can use it like so:

<AccordionListItem title={"List Item"}>
  <Text>Some body text!</Text>
</AccordionListItem>

The top bar of our list item will need to be wrapped in a TouchableWithoutFeedback since we want users to be able to open and close the list item by pressing it. This can contain the title we passed in via props.

<TouchableWithoutFeedback>
  <View style={styles.titleBackground}>
    <Text>{title}</Text>
  </View>
</TouchableWithoutFeedback>

Now let’s add a View for the list item body. This is the section that slides in and out when the header is clicked, and will contain our child component.

<View style={styles.bodyBackground}>
  {children}
</View>

Let’s see what we have so far:

Nothing revolutionary right? Bear with me, here’s where it gets interesting!

Now I’m hooked

Next up we’ll want to add a few React hooks to keep track of the state of our component. Here they are:

A state to keep track of whether the item is open or closed.
An Animated.Value to animate the opening and closing of the body section.
The height of the body section - this is what we will be changing in order to show and hide it.

Or in code form:

const [open, setOpen] = useState(false);
const animatedController = useRef(new Animated.Value(0)).current;
const [bodySectionHeight, setBodySectionHeight] = useState(0);

Note that we’re using useRef to create the animation object. Since our open state will persist across component refreshes, we want our animated value to do the same, otherwise they’ll get out of sync! useRef acts as a sort of state container for our Animated.Value, elevating it out of the component lifecycle.

Accordion time!

Now we need to tie our Animated.Value to the height of the body view. Our animated value can vary over whatever range we want but it’s easiest to pick a simple range like [0, 1]. Here, 0 maps to the list item being closed and 1 to it being open. As the animated value changes from 0 -> 1 we want the height of our view to go from 0 -> bodySectionHeight. To achieve this we can use the interpolate function:

const bodyHeight = animatedController.interpolate({
  inputRange: [0, 1],
  outputRange: [0, size.height],
});

Which simply maps the inputRange to the outputRange exactly as we described. When we change the value of the animatedController, the bodyHeight variable will scale this value relative to the height of our container. With all this set up, we can now write our toggleListItem() function, which will be called when we click on the item header.

We can think about what this function has to do as follows:

If the list item is open:
a. Change the animated value from 1 to 0.
b. Set the state to closed.
If the list item is closed:
a. Change the animated value from 0 to 1.
b. Set the state to open.

const toggleListItem = () => {
  if (open) {
    Animated.timing(animatedController, {
      duration: 300,
      toValue: 0,
    }).start();
  } else {
    Animated.timing(animatedController, {
      duration: 300,
      toValue: 1,
    }).start();
  }
  setOpen(!open);
};

I’ve set the duration to 300ms, but you can play around to find your preferred speed.

We’re getting pretty close. We’ll need to make our View an Animated.View in order to use React Native’s animation system, and then set the height of the animated view to be the bodyHeight variable.

<Animated.View style={[styles.bodyBackground, { height: bodyHeight }]}>
  {children}
</Animated.View>

Now for a sneaky step. We need to set an initial value for the height of our bodyView so we know what value to return when we open the list item. But, we can’t use the height of the animated view. This is because we are going to set the bodyHeight using the onLayout prop of View. For an animated view this is called many times while the view is animating! This is no good for us, we just want to set it once.

To fix this we can nest another regular view in our animated view which can have a constant height. We can then use this view’s height for the value of bodyHeight.

onLayout takes a function with an event parameter. We access the components height via event.nativeEvent.layout.height. So, putting it all together:

<Animated.View style={[styles.bodyBackground, { height: bodyHeight }]}>
  <View onLayout={setSize} style={styles.bodyContainer}>
    {children}
  </View>
</Animated.View>

Finally we need to set the overflow: “hidden” property on the bodyBackground style. This tells the children of the animated view what to do if they overflow the views box, which is exactly what will happen when the list item is closed. Setting this to hidden means they won’t be visible, perfect!

Realistic slide

What we have is nice, but it would be great if we could give the impression that the body of the list item is unfolding from the title bar. Fortunately there’s an easy way to achieve this! We can add the following properties to the bodyContainer style:

position: "absolute";
bottom: 0;

This ensures that the bottom of the inner view is always aligned with the bottom of the outer view, meaning the inner view should slide nicely up and down as the list item opens and closes.

Spinning arrows

Another nice feature would be to have an arrow indicator that rotates as the list item opens and closes. With our new knowledge of React Native animations, this should be easy! As our animated controller varies from 0 -> 1, we want the angle of our arrow to vary from 0 -> π radians or 0 -> 180° for regular folks. Using the same animated controller means it will always be in sync with the opening and closing. So we have:

const arrowAngle = animatedController.interpolate({
  inputRange: [0, 1],
  outputRange: ["0rad", `${Math.PI}rad`],
});

Then we can add our chosen arrow icon, within an Animated.View, into our title container view:

<View style={styles.titleBackground}>
  <View style={styles.titleContainer}>
    <Text style={styles.titleText}>{title}</Text>
    <Animated.View style={{ transform: [{ rotateZ: arrowScale }] }}>
      <DownArrowSVG width={20} height={28} />
    </Animated.View>
</View>

Here we have used the transform CSS property to pass in a z-angle of rotation for the view (in other words, the angle perpendicular to the z-axis - I had to work this out by trying rotateY and rotateX first!).

Take it Easing

One for the true perfectionists out there. React Native allows us to change the velocity of the animation via the easing parameter. This is a subtle change in our case, but it’s a good habit to get into as it can make a big difference in feel for some animations.

Most of the time I use the standard Material Design easing:

easing: Easing.bezier(0.4, 0.0, 0.2, 1);

Putting this in both our Animated.timing configs, we get our finished product!

Even after all that work our list item is less than 1kb in size once gzipped! I challenge you to find a library that’s smaller. We also have complete control over every aspect of our styling. Stick this in a FlatList component to create a nice scrollable list, or you can just stack them like regular components if you only need a few. Have fun with it!

Why it’s time to consider Azure to host your static application

Mon, 15 Jun 2020 00:00:00 GMT

The line between static & dynamic web applications is not what it used to be. With new frameworks like Gatsby or NextJS, the Jamstack is a strong option to get your application running in no time, whilst being able to scale it as needed.

When it comes to hosting your app, there are dozens of services available today, from Netlify to Firebase, but Microsoft has never really been one of those options (apart from hosting your app using an Azure Blob Storage) until last month, when they released their new service in preview, Azure Static Web Apps.

Here are the reasons why I think it is worth considering.

Get your app hosted & running in no time

If you haven’t used Azure before, you’d be surprised how easy it is to get started, and not only will you get set up in no time, but you’ll benefit from 12 months of Free Tier on most used services. Check it out here

Once you’ve picked your favourite frontend boilerplate and pushed it to Github, the only thing left to do is specify in your Azure Portal which branch you want to use. In less than a minute, your app will be hosted and available publicly.

In the background, Azure Static Web Apps service will generate & push a Github Action template to your repository, which will be triggered every time a branch is updated, creating for you multiple environments (even for each PR) without having to worry about CD. You can then enhance this action to add your own unit and integration tests.

Take advantage of the Microsoft Developer Experience

We have to admit that Microsoft have done an amazing job with VSCode. It is today, without any doubt, the most used IDE for web developers (haven’t tested it yet? check it out here https://code.visualstudio.com/ - In Theodo UK, we have decided to make it our default environment for web projects) And with such tools, the more popular they are, the more their ecosystem will become rich thanks to its community.

In our static web application example, it takes 2 clicks to download a LiveServer extension and have a server running your code locally, with hot reload. If you’re then planning to retrieve data in your app from an API, check out their extension which allows you to simulate your Azure Function running locally, test it, and deploy it all of that from your IDE.

Finally, you can write your app in TypeScript, framework designed by Microsoft, to type your JS code on your frontend and API, for a smooth and reliable experience.

Rely on Serverless to scale your app

If you’re not familiar with Serverless, we could summarise it this way: it is the concept of not provisioning servers, relying only on cloud services, to host your code and scale it up and down as needed automatically, greatly reducing the maintenance cost. (want to learn more? check out https://serverless-transformation.com)

As I mentioned just before, Azure Static Web Apps service allows you to use an API to retrieve data in your app. To do so, Microsoft give us the possibility to use Azure Functions, in JavaScript or TypeScript, to then be called directly from your frontend code. Once your environment variables defined, you just have to configure your SSL, your domain name and your app is good to scale!

Conclusion

Since Microsoft have gone down their open source journey, they are becoming a strong contender for a move to the cloud. Want to get started? Check out their quickstart

Don't overestimate the importance of performance when choosing a stack for your Web project

Thu, 14 May 2020 00:00:00 GMT

Understand how a suitable backend language can make a server handle higher loads and help reduce operating costs – or not.

A few months ago, a colleague of mine gave a talk detailing how great his recent experience with Go had been. As a side-note, he mentioned that Web applications written in Go were able to handle way greater loads than those powered by its mainstream Web competitors. This intrigued me: if this performance gap does exist, what is it due to, and is it significant enough for me, a Web developer, to actually care about it?

After a bit of research, I was confident I had a better understanding of the factors at play in the performance of Web servers. Still, I was lacking some quantitative insight that would allow me, at last, to decide whether taking these differences into account would help me bring more value to my next project. This is when I chose to run a small home-made benchmark comparing Node, PHP, and Go – three technologies we use at Theodo – that I will use to draw the final conclusions of this post.

What is performance anyway?

The first step of my investigation was to get sure of what I meant by performance. Although there is no single definition of this concept, I am confident that the following is relevant to my needs as a Web developer:

Performance is the ability of a server to handle a great number of requests during a given period of time.

Thus, a server that performs well according to this definition will be able to run on less powerful hardware, reducing operating costs and environmental footprint.

It is also correlated to other definitions of performance you may have thought about. For example, the faster a server can respond to a request, the more requests per minute it should be able to handle.

What a performant server is made of

The next thing to realize when trying to understand how a particular technology may influence the performance of a server is that during the whole process of serving an HTTP request, time dedicated by the CPU to executing code actually written by the Web developer or by the Web framework developers is often extremely small compared to:

Time spent performing heavy computations, such as hashing a password, manipulating images, etc.
Time spent waiting for network calls to terminate. Our server is most likely communicating with distant APIs and databases, or with the local file system. This takes way more time than executing a few thousand instructions on a CPU.

As counterintuitive as it may seem at first, heavy computations will not make much of a difference here. "Slower” interpreted languages will typically call well-optimized binaries under the hood to handle those kinds of tasks.

[Summary]:
   ticks  total  nonlib   name
      0    0.0%    0.0%  JavaScript
     72   63.7%  100.0%  C++
      1    0.9%    1.4%  GC
     41   36.3%          Shared libraries

What will make a difference on the other hand is the way the server uses the waiting times during the processing of a request to start handling other requests concurrently.

Scheduling how the CPU will handle numerous concurrent connections to the server is a complicated problem that can be addressed in a number of ways:

Typical PHP servers will let OS threads handle the task. To put it simply, threads are the mechanism that operating systems use to give us the feeling that they are processing several tasks in parallel. Thus, threads are also natural candidates for managing the concurrent processing of HTTP requests – each thread being responsible for a single connection – even though this is not what they were primarily designed for. This unfortunately comes with some performance issues: each new thread consumes quite a lot of memory, and time spent switching from the execution of a thread to another is also non-neglectable.
Technologies such as Node or Go will keep the processing of all the requests in a single or few threads but will use their own user-land mechanisms, such as an event-loop (Node) or Goroutines (Go), to address the scheduling issue. Although these two mechanisms will make Web developers write very different-looking code, they have in common that they do not suffer from the same overhead as OS threads do when opening a new connection or when switching from a connection to another.

Benchmark

We now know that thread-based Web servers should in theory be outshined by their counterparts that use a more suitable concurrency model. But is the performance gap wide enough to be of any importance to developers? Let’s find out!

To carry my little experiment, I set up a server and wrote three different routes in PHP, Nodejs, and Go (more details about versions and my exact protocol in appendix):

When route 1 is fetched, the server hashes a string then returns OK.
When route 2 is fetched, the server performs a call to a (very) distant server, waits for the response, then returns OK.
Route 3 is a mix between route 1 and route 2: when it is fetched, there is a 1% chance that the server hashes a string, else it performs a call to the distant server. It emulates a server that performs a lot of I/O (database queries for instance), and some CPU-intensive tasks (such as hashing passwords to identify users).

To compare performance, we’ll put some load on every route, one by one, for every language, and use the following comparison criterion:

Reference load = the maximum number of HTTP requests a server can handle in a minute, such as 95% of requests take less than 3 seconds.

Route 1 falls into the heavy computations category, so we expect to see little difference between all three technologies:

The small variations we observe can probably be explained by slight differences in bcrypt implementations. An interesting fact to notice here is that the reference load could have been predicted by dividing 60 seconds (1 minute) by the time it takes to complete a single isolated request (PHP case: 60 / 0.320 = 188). This means that our server is simply processing the requests one after the other, and has no opportunity to parallelize anything since there is no waiting time while handling a request.

So far, our results match the theory. Great!

Route 2 is much more interesting since there is now plenty of time for the server to parallelize requests while it is waiting for an API call to return. Let’s see what happens in this case:

We notice that the time it takes to handle a single isolated request is exactly the same no matter the language, which seems natural since the bottleneck of our route is a very long network call. However, huge differences arise when we start putting some load on the server:

Our Node server can handle about 20% more load than its PHP counterpart, which is significant, but definitely not as spectacular as the 1-to-4 ratio in RAM consumption. This seems to validate that thread-based servers are more memory-intensive than servers that use a more specialized concurrency model!
Meanwhile, our Go server is able to keep a low memory footprint while handling almost twice as many requests per minute than the former two. I’ll admit that I have yet to find a satisfactory explanation for this result, and if you do have one, I’d be very grateful if you could take the time to share it with me!

However, a real-world server could very well need to perform a few CPU-intensive operations among all these network calls, which is what I tried to emulate with route 3:

The most noticeable thing, in this case, is PHP’s memory consumption, and it is yet far from reaching any critical level. Reference loads on the other hand seem once again fairly similar.

Let's talk about money

All that is very interesting, but what matters in the end is how much money you will have to spend to run your application.

As an example, let's consider a large Symfony (PHP) project we are currently developing at Theodo. It handles about 50 million requests a day and runs on 32 machines with 8 cores and 32Gbytes of memory each. A similar infrastructure on AWS roughly costs $100,000 a year.

Our benchmark says that a more performant language could potentially divide infrastructure requirements by two. This would mean a $50,000 economy per year.

Conclusion

These results are telling me that choosing a suitable technology could have a measurable impact on operating costs if my service is mainly performing I/O, with close to zero CPU-intensive tasks. Otherwise, my application’s bottleneck will lie in the CPU-intensive code, and the technology I choose for writing my controllers will be of no importance whatsoever.

Let's also keep in mind that there are many more practical considerations that should be taken into account first when choosing a production-grade technology, as they may have a more significant impact on a project success. For example, a rich and mature ecosystem should come with a greater development speed, and should make it easier to find experienced developers to hire.

Note that Web performance in general should definitely be a concern for whoever wants to build a successful product! For that, you can check our performance experts at Theodo [FR].

Appendix: more details about the benchmark

The benchmark was run on an AWS EC2 t2 micro, Ubuntu 18.04.

Nginx 1.14.0, PHP-FPM 7.2
Node 12.14.0, managed by pm2 4.2.1 (and Nginx as a reverse proxy)
Go 1.13.4 (and Nginx as a reverse proxy)
I used loader.io as a load generator.
Code available here.

Measure the server-side impact of your application with PowerAPI

Thu, 14 May 2020 00:00:00 GMT

According to a report made by The Shift Project, the carbon footprint of the digital is estimated to exceed air travel with more than 4% of the total greenhouse gas emissions. This number is expected to double in the next five years!

Green IT principles aims to reduce the impact of the digital. As a developer, I wondered how these principles could be used for the web applications I worked on, and what could be improved.

I first looked at the definition of digital. It includes:

Networks (for instance wifi and 4G)
Data centers (used to host the servers and the data)
Terminals (smartphones, computers, TVs)
IoT (connected objects like a home assistant or a smart thermostat)

In this chart, the Shift Project shows that the energy consumption can be divided in two factors:

45% of the total emissions is dedicated to the production of digital equipment. To reduce the carbon footprint, it is important to keep a computer or a smartphone as long as possible. However, the average lifespan of a computer is estimated around 2 years. As developers, we can have an influence on the lifespan of our users' devices by making sure our applications are not too slow on old devices. Indeed, when you need 30 seconds to open most websites, your first thought usually is: "I really need to change my phone"!
55% of the emissions come from the usage of the equipment. We can have an impact on this energy consumption by designing sober and low-consuming applications.

So how can we measure the impact of a web application on our user’s device?

Some websites like Website Carbon ou EcoIndex (for french users) can estimate the client-side carbon footprint of a web application.
There is also the 1-byte-model which estimates the global carbon footprint. It takes into account the energy consumption of the data center, the network and the user device. A browser extension exists to see the impact of a website.

The data centers represents 19% of the digital energy consumption and some improvements in efficiency and performance can be found. However, there is no easy way to measure the carbon footprint of the server-side of an application. Therefore, it is complex to know where to begin to reduce the impact!

Example of a web application

At Theodo, we use an internal application to display the developers' current missions and availabilities. It is used by the sale representatives to see which developers are available and to staff them on new projects. One person has access to a CSV file online, and updates it several times a day. It is important to always have up-to-date data on the application in order to correctly staff the developers.

To update the application database, a script runs every 15 minutes: it imports the CSV file and parses it, then modifies the availability of the developers in the database. The file, which is quite large, is therefore imported 96 times a day. I had the intuition that it was consuming a lot of energy to run this script as many times, and I looked for a way to confirm or deny it.

I searched “How can I measure my server energy consumption” online and found out that simplest way was to plug a wattmeter into my computer, run the script locally and monitor the energy consumption. However, like most people, I don’t have a wattmeter at home, and I was not very keen on buying one for this unique usage.

I finally discovered a very useful tool: PowerAPI.

PowerAPI

PowerAPI is an open-source project developed by the Spirals research group (University of Lille 1 and Inria) in order to estimate the power consumption of a software, without any external device like a wattmeter. The calculation is made based on the analysis of the consumption of various hardware components (for instance CPU, memory, disk…). The objective is simple: design less energy-consuming software by easily monitoring their power consumption.

The power meter created by PowerAPI is composed of two components: the sensor and the formula

the sensor collects raw data from the hardware while the software is executed and stores it in a database.
the formula accesses this database, processes the raw data into a power consumption in Watt. The formula can store the results in the database or export it in a CSV.

As the sensor needs information from the Linux kernel, PowerAPI can only be used on Linux machines and not on a virtual machine.

Install PowerAPI

To install PowerAPI, you need:

Docker installed on your machine
a MongoDB database that can be accessed by your machine.

In my case, I decided to execute the project locally and deploy the MongoDB instance on my computer.

In the MongoDB database, I created a database (I decided to call it power-data) and two collections:

a collection to store the raw data from the sensor (in my example sensor-data)
a collection to store the computed results from the formula (formula-data)

You can either monitor the global power consumption of your machine, or monitor the power consumption of all Docker containers. As my project was using Vagrant, I decided to monitor the global power consumption of my computer. However, I recommend monitoring the Docker containers as it will reduce the “noise” created by your machine.

To install PowerAPI, the PowerAPI documentation will guide you. Once the MongoDB instance is created, the power meter can be deployed in two commands and a few minutes. Here’s a rapid recap, to measure the global power consumption:

Deploy the sensor: launch the first command to create the sensor Docker container.

docker run --privileged --net=host --name powerapi-sensor --privileged -td \
       -v /sys:/sys \
               -v /var/lib/docker/containers:/var/lib/docker/containers:ro \              
               -v /tmp/powerapi-sensor-reporting:/reporting powerapi/hwpc-sensor \        
               -n powerapi-sensor \
               -r "mongodb" -U "mongodb://ADDR" -D "power-data" -C "sensor-data" \
               -s "rapl" -o -e RAPL_ENERGY_PKG

Deploy the formula: launch the second command to create the formula Docker container.

docker run -td --net=host --name powerapi-formula powerapi/rapl-formula \
       -s \
       --input mongodb -u mongodb://ADDR -d power-data -c sensor-data \
       --output mongodb -u mongodb://ADDR -d power-data -c formula-data

The power meter is now ready, with two running Docker containers: the sensor and the formula.

Start measuring

If both Dockers are running, the power estimation is stored in the database. To stop monitoring, stop the Dockers.

The results

Once PowerAPI was deployed:

Step 1: I first started to monitor my computer energy consumption, to estimate what was the “noise” created by my machine.
Step 2: I launched the vagrant to start my application
Step 3: I ran the script 5 times in order to get average data.

The formula had stored all my power consumption data in the database. Now, I needed a way to display and analyse it.

How to process data from PowerAPI

First, I decided to use MongoDB Charts to display my results.

MongoDB provided me a simple interface to display the results written by the formula. I can for instance filter by date, to create a graph for a precise duration. On this chart, I displayed the energy consumption of my application during 1 minute.

However, in order to extract more information from the results, I decided to extract the data as a CSV, which can be done:

directly in the formula parameters by adding --output csv at the end of the command to launch the formula Docker
with a MongoDB command
via MongoDB Charts.

Once I had the data, I started to analyze it.

The results were given in Watts. When dealing with electric consumption, it is easier to use Watt-hours: for instance, you need 1 kWh to bake a cake in an oven during 1 hour. It means that your oven consumed 1000 Watts during 1 hour.

	Duration (s)	Average Power (W)	Consumption (Wh)
Step 1	225.5	0.81	0.05
Step 2	225.5	0.83	0.05
Import	225.5	5.47	0.34

The import lasted in average 225.5 seconds, about 3min45.

In Step 1, my computer alone consumed on average 0.81 W, i.e. 0.05 Wh during the imports.
In Step 2, my computer with the vagrant containing the application running used 0.83 W. Step 1 and Step 2 are almost equivalent in terms of Wh. In other words, running the application on Vagrant does not consume of lot of energy.
On average, one import used 0.34 Wh. The electric consumption almost did not vary between imports.

If I subtract the noise of the computer, an import alone consumes 0.29Wh. Imports are made 96 times a day, i.e. 35 064 times a year. It means the imports use each year around 10 kWh.

A Danish association, Tomorrow, created an interactive map to see in real time the countries' electricity production and its impact in terms of CO₂. Based on this website and on an 24-hours average, in France, 10 kWh corresponds to 3.5 kg of CO₂ emitted, as our energy comes mostly from nuclear. In the United States, 10 kWh corresponds to 11 kg CO₂.

If the server running my application was behaving exactly like my computer, it would use around 7 kWh a year.

To sum up, I estimated that the application was using 17 kWh each year, and 60% of this energy consumption is made by the imports. The rest is the energy consumes by the server, in the hypothesis of a physical server.

What can we usually do with 17 kWh? With 17 kWh, you can use your air-conditioning for 102 hours, cook 17 chickens in your oven or travel 8.5 km with a small electric car.

It is quite a lot for the server of a small web application.

What can be done to reduce the impact of this application?

I spoke with the developer and the person which was updating the CSV file, and we decided to:

take a quick action and rapidly stop importing the file outside work hours. Importing the file only between 9a.m. and 7p.m. on weekdays reduces the total consumption of the imports to 3 kWh a year, in other words a reduction of 41% of the electric consumption!
in the long term, create a button in the application, which can only be seen by the person updating the CSV file. He would use it when he updates the file, a few times a day. It will reduce the emissions of 95%.

What about the cloud?

Thanks to PowerAPI, I was able to estimate the impact of a costly feature on a physical server. I provided figures to help the project team make decisions to reduce the carbon footprint of the application.

However, this server runs on AWS, and PowerAPI cannot be used in a virtual environment in the cloud. In order to estimate the real impact, we need to find a methodology, using data from the cloud provider: for instance the amount of CPU used by the application.

I will try to see the real impact in the cloud of this feature in an upcoming article!

Flask Debugging in VS Code with Hot-Reload 🔥

Mon, 11 May 2020 00:00:00 GMT

I love using a debugger when I code. It allows me to quickly understand why something does not work as intended, but also to get a faster and deeper understanding of code I did not write.

Since I am so fond of using a debugger, when I started working on a Dockerized Flask application, my first online search was to find how to set one up for my application. But all the solutions I found had different flaws:

❌ The application port would be changed every time I started the application with the debugger
❌ Flask's wonderful hot-reload feature (the server restarting after saving changes to the code) was not supported
❌ The usage was clunky. To make the debugger work, I had to consistently add then remove multiple lines of work

So, I decided to craft my own debugger setup to fix all those problems.

Let's see how to setup a Dockerized Flask app with an efficient debugging flow 🎉!

🔧 Prerequisites

To follow this tutorial, you will only need the following installed:

VS Code
Docker
Docker Compose
The VS Code Python extension

Step 1: Docker setup

To follow this tutorial, make sure you have a Docker configuration similar to this:

# docker-compose.yml
version: "3.4"

services:
  flask-server:
    image: flask
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - 5000:5000
    volumes:
      - .:/app:cached

# Dockerfile
FROM python:3.8

EXPOSE 5000

# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE 1

# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED 1

# Install pip requirements
ADD requirements.txt .
RUN python -m pip install -r requirements.txt

WORKDIR /app

# Switch to a non-root user, please refer to https://aka.ms/vscode-docker-python-user-rights
RUN useradd appuser && chown -R appuser /app
USER appuser

Here we have setup a simple flask-server service that will run our Flask application inside a Docker container.

Step 2: Setup the debugger

VS Code configuration

The only configuration you will need is adding or modifying the .vscode/launch.json file:

// .vscode/launch.json
{
  "configurations": [
    {
      "name": "Python: Remote Attach",
      "type": "python",
      "request": "attach",
      "port": 10001,
      "host": "localhost",
      "pathMappings": [
        {
          "localRoot": "${workspaceFolder}",
          "remoteRoot": "/app"
        }
      ]
    }
  ]
}

Be sure to properly setup the pathMappings property. This will be used to link the files being executed in the Docker container to the files in your machine.
The .vscode folder stores all the project's VS Code configuration files.

Install the debugpy Python module

debugpy is a Python module that will allow you to spawn a debugger inside our Python code.

To install it, make sure to add debugpy to your requirements.txt:

flask==1.1.2
+debugpy

Not specifying the version (ie: not adding a line like this debugpy==1.0.0) will automatically install the latest version of the module.

Use debugpy to create a debug adapter instance

Create a debugger.py file in your application:

# debugger.py
from os import getenv

def initialize_flask_server_debugger_if_needed():
    if getenv("DEBUGGER") == "True":
        import multiprocessing

        if multiprocessing.current_process().pid > 1:
            import debugpy

            debugpy.listen(("0.0.0.0", 10001))
            print("⏳ VS Code debugger can now be attached, press F5 in VS Code ⏳", flush=True)
            debugpy.wait_for_client()
            print("🎉 VS Code debugger attached, enjoy debugging 🎉", flush=True)

Let's explain what is happening here:

Debug adapter logic

debugpy.listen(("0.0.0.0", 10001))

This will start the debug adapter that will listen for a client connection at the 0.0.0.0:10001 interface.

debugpy.wait_for_client()

This line will block program execution until a client (in our case, the client will be the VS Code debugger) is attached.

More logic for a better experience

if getenv("DEBUGGER") == "True":

We want the debugger to be spawned only if the DEBUGGER env variable is set to True. That way, we will still be able to run our application without it.

if multiprocessing.current_process().pid > 1:

In debug mode Flask uses a first process (with pid==1) to start child processes that handle connections. If the code below this line is executed by the main process, the debugging port is taken and subsequent child processes can't use the same port and are attributed a random port which prevents connections.

Instantiate the debugger

Given that your Flask application instance app is created in a file named app.py, call the initialize_flask_server_debugger_if_needed function inside it like this:

# app.py
from flask import Flask
from debugger import initialize_flask_server_debugger_if_needed

initialize_flask_server_debugger_if_needed()

app = Flask(__name__)

@app.route("/")
def home():
    return "Hello, Flask!"

Expose the debugging port

To be able to listen to the debug port of your Docker container, add the 10001 port to the list of exposed ports:

# docker-compose.yml
...
services:
  flask-server:
    ...
    ports:
      - 5000:5000
      - 10001:10001
    ...

Launch your application with a debugger 🎉

Everything is set up, now we only need to start our application!

Here are the docker-compose commands in a Makefile to launch your application with hot-reload using the flask executable.

# Makefile
## 🌶 flask and hot-reload
flask:
	docker-compose run --rm -e FLASK_APP=app.py -e FLASK_ENV=development --service-ports flask-server flask run --host 0.0.0.0

flaskdebug:
	docker-compose run --rm -e DEBUGGER=True -e FLASK_APP=app.py -e FLASK_ENV=development --service-ports flask-server flask run --host 0.0.0.0

The only difference between the two commands is the -e DEBUGGER=True parameter. This will set the DEBUGGER env variable to True inside your container, allowing the debugger code to be executed.

Here's a GIF to show the debugger in action:

To summarize, here are the steps to follow:

In a terminal, run make flaskdebug
When prompted ⏳ VS Code debugger can now be attached, press F5 in VS Code ⏳, press F5
Add a breakpoint to the line you want to debug by clicking left of its number
Call the corresponding route (localhost:5000/hello/flask in the GIF)
The debugger stops the code at your breakpoint, happy debugging 🎉

Hot-reload in action 🔥

Now that it's easy to use the debugger, let's see how the application hot-reload feature is supported.

As you can see in the GIF, after I saved modifications to a file:

The application restarts
The debugger is detached
I am prompted ⏳ VS Code debugger can now be attached, press F5 in VS Code ⏳ again
After I pressed F5, the application is now running with the new changes and the debugger is attached

Tips & tricks

Use the Debug Console to code on the fly

Whenever you are stopped at a breakpoint in your code, VS Code's Debug Console acts as a Python interactive console with the full current context of your code.

Here you can see that I was able to do the following:

Import date from the datetime module
Re-assign the name value to date(3000, 1, 1)
The route response is correctly updated with the modifications

The Debug Console is a very useful tool to quickly test edge cases and find fixes!

Running the app with gunicorn

To use gunicorn instead of flask to run the app, use these commands:

## 🦄 gunicorn and hot-reload
gunicorn:
	docker-compose run --rm --service-ports flask-server gunicorn --reload --bind 0.0.0.0:5000 app:app

gunicorndebug:
	docker-compose run --rm -e DEBUGGER=True --service-ports flask-server gunicorn --reload --bind 0.0.0.0:5000 --timeout 3600 app:app

Running the app as a top-level script

If you need to run your app as a top-level script, ie:

The app is run with python app.py

# app.py
...
if __name__ == "__main__":
  app.run("0.0.0.0", debug=True)

Be sure to make this modification:

if __name__ == "__main__":
    from debugger import initialize_flask_server_debugger_if_needed

    initialize_flask_server_debugger_if_needed()

    app.run("0.0.0.0", debug=True)

Thanks to this, if you want to use the app instance in another file:

# another-python-file.py
from app import app

The code initiating the debugger will not be executed when the import is made.

Wrap-up 🌯

I showed you how to resolve the three problems presented I encountered trying to setup a debugging flow in a Dockerized Flask application:

✔️ The application port does not change
✔️ Flask's wonderful hot-reload feature is supported
✔️ The debugger is easy to use

I created a repository containing a minimal application with everything shown in the article.

I hope that this article helped you setup a (better!) debugging flow of your Dockerized Flask application 💪.

You have a question? No problem, feel free to either contact me:

📧 by email
🐦 on twitter
🐙 by leaving an issue or a pull request on the tutorial repository
💬 by leaving a comment below

I will be very happy to discuss this tutorial with you 😃

Stop losing data when writing Django migrations !

Thu, 07 May 2020 00:00:00 GMT

Saying that database structure is important is sort of an obvious statement. That is of course if you decided to use a structured database technology. But in that case, you want your database structure to be the closest to your functional and business needs, and the tightest in order to be able to rely on that structure.

However, when I was working on Django projects, I sometimes found it painful to change the data structure when I needed to. However, I discovered a very useful tool offered by Django migrations : the RunPython command.

Simple migration

Let’s take a simple example and assume that we have a single model called Order inside an orders application.

We would then have the following models.py file:

from django.db import models


class Order(models.Model):
    reference = models.CharField(max_length=8)
    amount = models.DecimalField(max_digits=8, decimal_places=2)
    creation_date = models.DateField(auto_now=False, auto_now_add=False)
    due_date = models.DateField(auto_now=False, auto_now_add=False)
    customer_name = models.CharField(max_length=50)
    customer_address = models.CharField(max_length=50)
    customer_city = models.CharField(max_length=50)
    customer_zip_code = models.CharField(max_length=50)

    def __str__(self):
        self.reference

Running python manage.py makemigrations will produce the following migrations file, called 0001_initial.py :

from django.db import migrations, models

class Migration(migrations.Migration):

    initial = True

    dependencies = []

    operations = [
        migrations.CreateModel(
            name="Order",
            fields=[
                (
                    "id",
                    models.AutoField(
                        auto_created=True,
                        primary_key=True,
                        serialize=False,
                        verbose_name="ID",
                    ),
                ),
                ("reference", models.CharField(max_length=8)),
                ("amount", models.DecimalField(decimal_places=2, max_digits=8)),
                ("creation_date", models.DateField()),
                ("due_date", models.DateField()),
                ("customer_name", models.CharField(max_length=50)),
                ("customer_address", models.CharField(max_length=50)),
                ("customer_city", models.CharField(max_length=50)),
                ("customer_zip_code", models.CharField(max_length=50)),
            ],
        ),
    ]

Let's break down what's happening in this migrations file. The migration operation is represented as a python class, with 3 attributes:

initial states that this migration is the first of its Django app;
dependencies list all the migrations that need to be applied before this one can be applied. Here it is empty since the migration is the first of its Django app;
operations is the most important part. It is a list of actions that Django will apply to the database. Here the database was empty, so we created the model from scratch with the CreateModel method.

The python manage.py makemigrations performs well when the operations to perform are simple enough. Renaming a model, or renaming a field will be treated as equivalent database operations for example.

After applying the migration, let's create a few entities in the database.

Complex structure migration

Let's start again from our previous example and observe something that you may already have noticed. The 4 fields customer_name, customer_address, customer_city, and customer_zip_code are functionally attached to the Order entity, while they represent another physical entity, a customer. And that is fine as long as there is no duplication. But now let's imagine that for business purposes, you need to abstract a Customer entity to which the Order entities are linked.

Your models.py file would then be:

from django.db import models

class Customer(models.Model):
    name = models.CharField(max_length=50)
    address = models.CharField(max_length=50)
    city = models.CharField(max_length=50)
    zip_code = models.CharField(max_length=50)

    def __str__(self):
        return self.name

class Order(models.Model):
    reference = models.CharField(max_length=8)
    amount = models.DecimalField(max_digits=8, decimal_places=2)
    creation_date = models.DateField(auto_now=False, auto_now_add=False)
    due_date = models.DateField(auto_now=False, auto_now_add=False)
    customer = models.ForeignKey(Customer, on_delete=models.PROTECT)

    def __str__(self):
        return self.reference

This configuration now fits our functional needs. Let's try and generate the migration !

> You are trying to add a non-nullable field 'customer' to order without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
 2) Quit, and let me add a default in models.py

There is a problem however. Django recognizes that we are trying to create a customer field on the Order model, which can't be None but is not defined. So it wants to add a non-null value to all the existing entries in the database, and asks us if we want to provide it now or want to set it in the Order model.

However, none of these options satisfy us. We don't want our orders to have customers that are either None or a default customer, which would lose all the existing data.

Granted, we could also execute this migration, then manually set the customer attribute of all the Order model, but then this forces you to execute this same action on all of your environments. That would also make the migration quite difficult to rollback.

Fortunately, Django comes with an built-in solution to deal with this limitation.

Generating a custom migration

First of, let's start by generating an empty migration that we will then edit.

> python manage.py makemigrations orders --empty

The generated migration file will look like this

from django.db import migrations

class Migration(migrations.Migration):

    dependencies = [
        ('orders', '0001_initial'),
    ]

    operations = [
    ]

We then need to build our custom migration. It needs to have 6 steps :

Create the new Customer model in the database, with all the necessary fields.
Create a nullable customer foreign key on Order
Set the Order fields to transfer as nullable
Transfer the data from the Order model to the Customer model.
Set the new customer field on Order as non-nullable
Remove the old fields from the Order model.

The Django documentation explains all the database operations needed.

Let's write this migration :

from django.db import migrations, models
import django.db.models.deletion

class Migration(migrations.Migration):

    dependencies = [
        ("orders", "0001_initial"),
    ]

    operations = [
        # step 1: add the new Customer model
        migrations.CreateModel(
            name="Customer",
            fields=[
                (
                    "id",
                    models.AutoField(
                        auto_created=True,
                        primary_key=True,
                        serialize=False,
                        verbose_name="ID",
                    ),
                ),
                ("name", models.CharField(max_length=50)),
                ("address", models.CharField(max_length=50)),
                ("city", models.CharField(max_length=50)),
                ("zip_code", models.CharField(max_length=50)),
            ],
        ),

        # step 2: add the nullable foreign key field `customer` to Order
        migrations.AddField(
            model_name="order",
            name="customer",
            field=models.ForeignKey(
                null=True,
                on_delete=django.db.models.deletion.PROTECT,
                to="orders.Customer",
            ),
        ),

        # step 3: set the order fields as nullable
        migrations.AlterField(
            model_name="order",
            name="customer_address",
            field=models.CharField(null=True, max_length=50),
        ),
        migrations.AlterField(
            model_name="order",
            name="customer_city",
            field=models.CharField(null=True, max_length=50),
        ),
        migrations.AlterField(
            model_name="order",
            name="customer_name",
            field=models.CharField(null=True, max_length=50),
        ),
        migrations.AlterField(
            model_name="order",
            name="customer_zip_code",
            field=models.CharField(null=True, max_length=50),
        ),

        # step 4: transfer data from Order to Customer
        ...

        # step 5: set the `customer` field as non-nullable
        migrations.AlterField(
            model_name="order",
            name="customer",
            field=models.ForeignKey(
                null=False,
                on_delete=django.db.models.deletion.PROTECT,
                to="orders.Customer",
            ),
        ),

        # step 6: remove the old Order fields
        migrations.RemoveField(model_name="order", name="customer_address",),
        migrations.RemoveField(model_name="order", name="customer_city",),
        migrations.RemoveField(model_name="order", name="customer_name",),
        migrations.RemoveField(model_name="order", name="customer_zip_code",),
    ]

In this migration, step 2 and 3 basically loosen the data structure, preparing it for the data transfer. Then step 5 and 6 tighten it again !

Let's now dive in step 4, where the magic happens !

Write the data transfer function

In order to perform this operation, we are going to use the RunPython migration operation. What it basically does is execute a python function using the ORM.

Its syntax is the following :

# step 4: transfer data from Order to Customer
migrations.RunPython(order_to_customer, reverse_code=customer_to_order)

In this step of the migration, we specify two functions :

order_to_customer will be run when running the migration ;
customer_to_order will be run when reversing the migration.

def order_to_customer(apps, schema_editor):
    Order = apps.get_model("orders", "Order")
    Customer = apps.get_model("orders", "Customer")
    for order in Order.objects.all():
        customer, _ = Customer.objects.get_or_create(
            name=order.customer_name,
            address=order.customer_address,
            city=order.customer_city,
            zip_code=order.customer_zip_code,
        )
        order.customer = customer
        order.save()

def customer_to_order(apps, schema_editor):
    Order = apps.get_model("orders", "Order")
    for order in Order.objects.all():
        order.customer_name = order.customer.name
        order.customer_address = order.customer.address
        order.customer_city = order.customer.city
        order.customer_zip_code = order.customer.zip_code
        order.save()

These functions look a lot like what you would write to manually transfer the data from one model to the other. However there is one trick. It isn't possible to import the models normally with from orders.models import Order, Customer. Instead, we need to use a versioned model passed to the migration function by Django.

Running the migration

After running this migration via python manage.py migrate, we can check that the migration has created the Customer instances and linked them to the correct Orders.

Customers

Orders

Et voilà ! This migration is now operational. It can also be easily rolled back with python manage.py migrate orders <previous_migration_id>. The data will then be transferred back to the Order model.

Conclusion

You now have a way to stop worrying about losing data when migrating your database with Django! The Django ORM is a great tool and although it does the job most of the time, understanding what happens under the hood is a great way of making your life easier when dealing with migrations.

Sources

Django documentation on migrations:
- https://docs.djangoproject.com/en/3.0/howto/writing-migrations/
- https://docs.djangoproject.com/en/3.0/ref/migration-operations/
Link to the Github repo with the example https://github.com/fargito/django-runpyton

How to Remain Agile with DynamoDB

Tue, 05 May 2020 00:00:00 GMT

Amazon DynamoDB is built to deliver single-digit millisecond performance at any scale. It is built to store Terabytes of data. It is built to support Amazon's Cyber Monday traffic.

However, this scalability comes with an overhead. No matter what resource you look up, you will be constantly reminded that "You Must Know Your Access Patterns In Advance". Amazon Marketplace had been running for years before they migrated to DynamoDB so they were well equipped to know how they needed to access their data and fully reap the rewards of a NoSQL setup.

In contrast, a new startup with an exciting new product wants to build an MVP quickly. They don't know how their app will grow over time. They want to be agile and react to their users to develop their product accordingly. They can't know all their access patterns in advance.

This no doubt poses an issue, but not an insurmountable one.

Being able to map your access patterns in as much detail as possible will translate to a smooth DynamoDB adventure. However, building a startup is about disrupting the market so you can't be afraid of a little turbulence along to way to reach your goal.

If you use DynamoDB from the start you will never have to worry about the scalability of your product. If your app gets a massive social media spike, you won't have to scramble to prevent your site going down. And reaching that scale is what all startups aspire to so how can we build towards this while limiting the turbulence?

Check out my full article on our Serverless Transformation Blog.

Why You Need a Framework When Doing Serverless

Mon, 27 Apr 2020 00:00:00 GMT

We are proud to have been selected as a Serverless Framework official partner. It rewards our investment in the technology and reflects our strong belief that this framework is currently the best in the market.

Using a Framework When Going Serverless is a Must

Recently, one of our clients’ technical advisors shared with us the challenges that he had encountered when testing serverless a few years ago. As he started to use AWS Lambda functions, he quickly lacked visibility of the infrastructure design and struggled to figure out which version of the code was running in production. Fortunately, these issues are now solved by frameworks for serverless applications.

At Theodo, we began leveraging serverless functions a few years ago for projects requiring asynchronous workloads. To avoid the aforementioned pitfalls, we looked into tools that would allow us to be more efficient in managing our serverless projects. The most popular tool was, and remains today, the Serverless Framework.

The main advantage of the Serverless Framework is to allow developers to write to one single repository containing:

The code of the business logic run by their serverless functions,
All the files that describe their application’s infrastructure, including:
- Database services like DynamoDB,
- Authentication services like Cognito or Auth0,
- Notification services like AWS Simple Notification Service or Twilio,
The scripts they need to easily deploy their application in one command to any environment,
Tools allowing developers to emulate services like DynamoDB in their local development environment. Other things that the Serverless Framework achieves is the deployment of a new stack with the whole infrastructure in a single command, allowing to run automated tests consistently on a ephemeral — hence cheap — production-like environment. This further stresses the productivity gains and cost reductions observed by companies embracing serverless.

Why We Chose to Work With the Serverless Framework

Its Large Community Guarantees its Relevance

The suite of serverless services offered by cloud providers is constantly expanding. Therefore, the value of a serverless application framework lies in the ability of its community to stay current.

The Serverless Framework has the largest community among its competitors, and it provides many plugins that gradually add support for new services as they are released. You can read about a few of our favorite tools for this framework in this article.

Its Independence From Cloud Providers

Other frameworks, like AWS’ own Serverless Application Model (or SAM), are open-sourced and maintained by cloud providers themselves — offering early support for their newest features and services. The purpose of some third-party frameworks, however, is to enable developers to deploy their serverless applications to any cloud provider. As the concerns about vendor lock-in grow, the appeal of using a cloud-agnostic framework becomes stronger. It allows you to move your infrastructure to another cloud provider if necessary.

The development of a reliable and powerful framework is a daunting and ambitious task, but we believe in the purpose of the Serverless Framework and we enjoy using it daily on many of our projects. We contribute to the community, for example with sls-dev-tools, our open-source project that gives developers visibility on their serverless application directly from the command line.

With close to sixteen million downloads and hundreds of thousands of weekly deployments, Serverless Framework has become the benchmark in the serverless ecosystem — and this is only the beginning.

Want to know more about how Serverless can help your company? I'd be happy to help. Please reach out here!

Reduce redux boilerplate with redux-toolkit

Mon, 20 Apr 2020 00:00:00 GMT

You're looking for an easier way to write your reducer, aren't you ?

When writing plain redux, you have to define an action, an actionCreator and a reducer to update the store of your app. The reducer is in most cases a big switch to handle all the actions dispatched by actionCreators in your app. This leads to a lot of boilerplate and make the code more difficult to understand. There are many librairies out there aiming to reduce this boilerplate.

At Theodo, we used typesafe-actions to reduce redux boilerplate. The redux team recently recommended @redux-toolkit as the "official recommended approach for writing Redux logic". Thus, I tested it and wrote this article to show you the advantages of such librairies through a simple example.

The example

The example consist in changing the owner of a car :

There is one car and 2 possible owners ("JOHN" or "JANE").

When you click on one of the owner, the car owner in the redux store is changed to match the owner clicked.

Plain redux (28 lines)

With plain redux as on the code below, there are :

an action (UPDATE_OWNER),
an actionCreator (updateOwner)
a reducer (itemReducer) to handle all the actions (codesandbox).

The first simplification external librairies provide is to get rid of the action and avoid the switch in the reducer function.

Both typesafe-actions and redux/toolkit give you two functions to simplify action and reducer : createAction and createReducer

We do not need to keep the UPDATE_OWNER action type anymore in an variable to handle it in the reducer.

The switch disapears

Code is easier to read and understand

Typesafe action (21 lines ~ -25%)

With createAction and createReducer functions there is no action constant UPDATE_OWNER and no verbose switch in our reducer anymore : the reducer reacts directly to the actionCreator updateOwner (codesandbox)

Type-safe action provides also a clearer way to write the reducer using the chain API (codesandbox)

You can even combine actions easily and/or chain action handlers:

const counterReducer = createReducer(0)
  .handleAction([add, increment], (state, action) =>
    state + (action.type === 'ADD' ? action.payload : 1)
  )
  .handleAction(add, (state=> action) => ...);

The code is clearer and easier to understand

Remarks :
- @reduxjs/toolkit provides also a `createAction` function but used in a slightly different manner see here.
- @reduxjs/toolkit `createReducer` function is used as typesafe-action

@redux/toolkit (18 lines ~ -35%)

@reduxjs/toolkit goes further and provides a function createSlice, allowing actions and reducers to be created in the same function. It's very powerfull, give it a try codesandbox.

Moreover Redux toolkit is shipped with ImmerJS, "a tiny package that allows you to work with immutable state in a more convenient way".

No need to duplicate the state with spread operators everywhere: You can directly update the state and ImmerJs will take into account the changes and create a new state from the precedent state + the changes you made. Have a look again to the reducer function 👇🏽

const { actions, reducer: itemReducer } = createSlice({
  name: "items",
  initialState: initialItemState,
  reducers: {
    updateOwner: (state: ItemState, { payload: { user, item } }: UpdateOwnerAction) => {
      // This long code with many ... is replaced by one line thks to ImmerJS
      // return {
      // ...state,
      // [item.id]: { ...state[item.id], owner: user }
      // }
      state[item.id].owner = user;
    }
  }
});

This makes reducer and actions easier to understand. Goodbye "desctructuration ..." 😍.

⚠️ Redux teams recommends to read ImmerJs pitfalls before using it, so read it 😉

In the end of March 2020, the redux toolkit team added 2 api functions (createEntityAdapter & createAsyncThunk) to simplify reducer logic.

In the last example I use the createEntityAdapter function : the entityAdapter "generates a set of prebuilt reducers and selectors for performing CRUD operations on a normalized state structure."

After normalizing my itemState and creating the itemsAdapter, I can simplify my reducer function to use the generic entityAdapter.updateOne to handle the ownerUpdate action (check full code here).

This gives you a very fast way to write your reducer logic.

Have a look at redux-toolkit doc to fully understand the createEntityAdapter API

Conclusion :

Reasons to use typesafe-actions

If you have a lot of actions to handle and one reducer has to react to several actions then typesafe-actions chaining will be right for you

⚠️ Typesafe-actions maintainer is currently updating their API to simpllify the syntax. If you want to contribute. ⚠️

Reasons to use redux/toolkit

Easier to write and understand reducers with ImmerJs
Slicers really concise
The API encourages you to use a normalized state and provides you with practical CRUD function to update data in your store

As redux/toolkit is the recommended library by the redux team, you should give it a try. Slicers + ImmerJS = 🔥 : your code is shorter and easier to understand. Moreover the createEntityAdapter will help you to normalize your state and write a new reducer faster thanks to the many CRUD functions provided.

Demystifying Building Native Modules for React Native

Fri, 17 Apr 2020 00:00:00 GMT

You’ve just released a new native SDK. Could you bring more business to your company by releasing on a cross-platform development framework?

Maybe you’re a developer, longing for a native module of an open-source platform API (or even the third-party SDK above 🙄) to be released. Could you fast-track the development yourself?

You could build your own module. With React Native, this becomes a straightforward process.

Let’s talk about React Native for a second.

Why React Native

For those unfamiliar with React Native: it is a cross-platform development framework allowing you to create native apps for iOS and Android. Using React, you can maintain two platforms using the same codebase, allowing faster development, iteration and knowledge-sharing.

With this framework, we have two sides; JavaScript and Native. Between the two is a bridge, allowing bidirectional and asynchronous communication. This is the power of React Native, on top of a multitude of other benefits.

In 2020, we see the likes of Facebook, Bloomberg and Shopify [1] using React Native to develop their mobile applications, amongst others in the Fortune 500. With over 11 million Javascript developers [2] and more companies switching to React Native, releasing your technology for React Native could bring more growth.

We’ve seen other cross-platform development frameworks like Flutter and Xamarin climb the ranks over the last few years, but React Native is still gaining popularity and ever-increasing its performance as it gets more mature.

So hopefully that's convinced you to consider releasing a Native Modules for React Native if you're releasing Native SDKs. Let’s break down how you’d do that.

Native to React Native

As previously mentioned, communication from the native world to React Native is asynchronous. What this means is that any values must be sent through asynchronous callbacks, promises and events, each sent on a batched message queue. This is the architecture as of April 2020; there is currently a re-architecture of the React Native internals and due to be released mid-2020 [3].

On the native side:

The main thread is responsible for the UI.
The shadow queue is responsible for layout calculations.
Each native module has its own thread (Android shares a thread pool)

On the JavaScript side:

JavaScript VM thread which runs the bundled JS code and sends instructions to the native threads via the bridge.

The two sides communicate over the bridge using the message queue.

JavaScript knows about your native modules at runtime - there’s a JSON representation of each Native Module (consisting of module id, method id and arguments) and we can call methods on the Native Module this way.

For the other direction, Native to JavaScript, we can use promises, callbacks and event to transfer data.

To read more about the React Native internals and the upcoming re-architecture, read this article [4].

Module structure

To get started building your native module, there are a number of tools to help set up the skeleton of your project. For example:

We like Bob, as your library will then come pre-configured with TypeScript and support for Kotlin and Swift. If you've got any other favourites, let us know!

For project structure, if you’ve chosen Bob, the library should now have the following project structure:

- /ios
- /android
- /src
    - index.tsx
- /yourExampleApp
    - package.json
    - ...
- ...

In the iOS and Android folders, you’ll want to have your native SDK code - either available in your project or using git submodules to keep the version control.

Exposing iOS Modules

On the iOS Side, there are two languages at play: Objective-C and Swift. You can write the native module entirely in Objective-C, or write methods in Swift and expose them to Objective-C by using the @objc attribute.

The steps are then as follows:

Create a Bridging Header file (Objective-C, .h file) - Here you'll import:

#import "React/RCTBridgeModule.h"
#import <React/RCTLog.h>
#import <React/RCTEventEmitter.h>
#import <React/RCTConvert.h>

Create a Bridging Command file (Objective-C, .m file), expose modules with RCT_EXPORT_MODULE and Expose methods with RCT_EXPORT_METHOD

#import <React/RCTBridgeModule.h>
#import <React/RCTEventEmitter.h>
#import "MyModule.h"

@interface RCT_EXTERN_MODULE(MyModule, NSObject)

RCT_EXTERN_METHOD(myMethodWithNoParams)

RCT_EXTERN_METHOD(myMethodWithAPromise:(NSString)input
resolver:(RCTPromiseResolveBlock)resolve
rejecter:(RCTPromiseRejectBlock)reject)

+ (BOOL)requiresMainQueueSetup
{
    return YES;
}

@end

Define event emitters, implementing the Delegate pattern.

#import <React/RCTBridgeModule.h>
#import <React/RCTEventEmitter.h>
#import <React/RCTConvert.h>

@interface RCT_EXTERN_MODULE(MyModuleEmitter, RCTEventEmitter)

// Create a singleton for the EventEmitter class

- (id)allocWithZone:(NSZone *)zone {
 static MyModuleEventEmitter *shared = nil;
 static dispatch_once_t onceToken;
 dispatch_once(&onceToken, ^{
 shared = [super allocWithZone:zone];
 });
 return shared;
 }

- (BOOL)requiresMainQueueSetup
 {
 return YES;
 }

@end

For Swift, a good pattern to use is the Observer-Command-Emitter pattern:

This ties together your observer which implements the delegate, the event emitter which triggers events and the commands which will be the methods to expose to JavaScript. In the above code snippets, you've seen the Objective-C .m files for the above.

Exposing Android Modules

The steps for Android are similar:

Create a ReactPackage to declare the modules to expose:

package com.mymodule

import java.util.Arrays

import com.facebook.react.ReactPackage
import com.facebook.react.bridge.NativeModule
import com.facebook.react.bridge.ReactApplicationContext
import com.facebook.react.uimanager.ViewManager

class MyPackage : ReactPackage {
override fun createNativeModules(reactContext: ReactApplicationContext): List<NativeModule> {
return Arrays.asList<NativeModule>(MyModule(reactContext))
}

   override fun createViewManagers(reactContext: ReactApplicationContext): List<ViewManager<*, *>> {
       return emptyList<ViewManager<*, *>>()
   }

}

Create your module extending ReactContextBaseJavaModule to declare the methods to expose using @ReactMethod . Implement listeners to emit events.

package com.mypackage

import com.facebook.react.bridge._
import com.facebook.react.modules.core.DeviceEventManagerModule.RCTDeviceEventEmitter
import java.util.\*

enum class MyModuleError(val errorCode: String, val message: String) {
 INPUT_NOT_RECOGNISED( "INPUT_NOT_RECOGNISED", "Input was not of the correct format."),
}

class MyModule(reactContext: ReactApplicationContext) : ReactContextBaseJavaModule(reactContext), MyListener {
 companion object {
   const val MODULE_NAME = "MyModule"
 }

 override fun getName(): String {
   return MODULE_NAME
 }

 private fun createErrorHandler(promise: Promise): (TokenError) -> Unit {
   return fun(error: TokenError) {
     promise.reject(error.errorCode, error.message)
   }
 }

 @ReactMethod
 fun myMethodWithNoParams() {
    doSomething();
 }

 @ReactMethod
 fun myMethodWithAPromise(input: String, promise: Promise) {
   val handleError = createErrorHandler(promise)

   doSomethingWithInputSucceeds(input)?. let {
     promise.resolve("MY_METHOD_SUCCESS")
   } ?: kotlin.run {
     handleError(ModuleError.INPUT_NOT_RECOGNISED)
   }
 }

 override fun onEventSuccess(result: String, message: String) {
   val body: WritableMap = Arguments.makeNativeMap(mapOf(
     "result" to result,
     "message" to message
   ))
   this.reactApplicationContext.getJSModule(RCTDeviceEventEmitter::class.java)
     .emit("MyModule/eventSucceeded", body)
 }

 override fun onEventFailure(error: String, message: String) {
   val body: WritableMap = Arguments.makeNativeMap(mapOf(
     "error" to error,
     "message" to message
   ))
   this.reactApplicationContext.getJSModule(RCTDeviceEventEmitter::class.java)
     .emit("MyModule/eventFailed", body)
 }
}

Your new module

Now you’ve exposed your native modules and methods to React Native and can continue to build on top of them. What’s left is to import them from NativeModules from react-native and you’re ready to create your app:

In /src of your Native Module:

import {
  NativeModules,
  DeviceEventEmitter,
  NativeEventEmitter,
} from "react-native";

export const { MyModule, MyModuleEventEmitter } = NativeModules;

Now you're ready to go. Just yarn add react-native-myModule and import the above MyModule and emitters to get going.

To launch your development experience and quality to the next level:

Add a React Native deployment pipeline - https://blog.theodo.com/2019/04/react-native-deployment-pipeline/
Add end-to-end tests with Detox - https://github.com/wix/Detox
Add support for offline - https://dev.to/reactnativeradio/rnr-157-building-great-offline-ready-apps-in-react-native-with-josh-warwick

What a great development set up!

Summary

Once that’s all ready, your module is tested, you’re ready to publish. Very straightforward with npm publish[5].

There you have it. A native module which can now be released to the JavaScript tech community - perhaps bringing in more customers for your business or fast-tracking your next development track.

Hopefully this article gave you an overall idea of where to start building your bridge module. If you have any questions, let us know, we'd be happy to help!

For more resources, the following are some good resources:

Sources + links:

How to convert an imperative library to a declarative React component: the Sketchfab Viewer

Fri, 10 Apr 2020 00:00:00 GMT

I love using React for multiple reasons. I like the writing style, I like the tooling, I like the ecosystem.

But sometimes you run into a library that doesn't have a React adapter. It can either be pretty old, kind of niche, closed source, etc.

In this series of articles, we’ll look into how to convert an existing imperative JS library into a declarative React component.

We’ll use the Sketchfab Viewer as an example. Sketchfab develops an amazing 3d suite that includes a very configurable web viewer for use with their models.

You can see that their doc is pretty extensive, if a bit messy. It includes a lot of examples and it took me a long time to run into a use case that was not already covered. Yet you can also see that they use an imperative style of JS that doesn’t fit well into the React patterns.

api.addEventListener("viewerready", function () {
  console.log("Viewer is ready");
});

We are going to build a React function component on top of that library. It will accept a config object to declaratively change a color on the model. It will also expose a callback to react to a click event on a part of the model. And we'll use hooks because they're fun!

You can find the final code for this part here.

Part 1: Initializing and exposing the api object
Part 2: Working on it!

Initializing and exposing the api object

Throughout this project we'll work with this fancy chair, courtesy of the official Sketchfab examples.

Start the project with create-react-app

First things first, let's generate a basic CRA project:

npx create-react-app sketchfab-react

In src/, add a Viewer.jsx file and create an empty component inside.

import React from "react";

export const Viewer = () => <></>;

Add that component inside App.jsx

import React from "react";
import "./App.css";
import { Viewer } from "./Viewer";

function App() {
  return (
    <div className="App">
      <Viewer />
    </div>
  );
}

export default App;

Now let's import the library. Sketchfab provides a package install through npm but that's not always the case for other libraries, so we'll include the source in the head of public/index.html.

<script
  type="text/javascript"
  src="https://static.sketchfab.com/api/sketchfab-viewer-1.7.1.js"
></script>

This import will make window.Sketchfab available globally. If you're using Typescript, you will need to ignore it.

Render the Sketchfab Viewer in an iframe

According to the docs, we need to create an iframe and give it to the Sketchfab constructor to render the Viewer.

The way to do this in React is, in Viewer.jsx:

import React, { useEffect, useRef } from "react";

// Our wonderful chair model
const MODEL_UID = "c632823b6c204797bd9b95dbd9f53a06";

export const Viewer = () => {
  // This ref will contain the actual iframe object
  const viewerIframeRef = useRef(null);

  const ViewerIframe = (
    <iframe
      // We feed the ref to the iframe component to get the underlying DOM object
      ref={viewerIframeRef}
      title="sketchfab-viewer"
      style={{ height: 400, width: 600 }}
    />
  );

  useEffect(
    () => {
      // Initialize the viewer
      let client = new window.Sketchfab(viewerIframeRef.current);
      client.init(MODEL_UID, {
        success: () => {},
        error: () => {
          console.log("Viewer error");
        },
      });
    },
    // We only want to initialize the viewer on first load, so we don't add any dependencies to useEffect
    []
  );

  return ViewerIframe;
};

And you should have the viewer displaying!

Expose the api object to access it imperatively

One of the great things about the Viewer API is that it allows us to transform our model from the browser. Here, we'll try to make the chair red.

First we need to extract the api object and store it. We could put it in the state of our Viewer component, but we'll want to use it higher up in the tree, so we will pass a ref as prop and give it the api.

export const Viewer = ({ apiRef }) => {
	// ...
        success: api => {
          // Store the initialized api inside the provided ref
          apiRef.current = api;
        },
	// ...
};

Now, let's get it inside App.

function App() {
  const apiRef = useRef(null);

  return (
    <div className="App">
      <Viewer apiRef={apiRef} />
    </div>
  );
}

To check that it works, let's try to change the color of the background. There's a method for that!

Let's add this code to App.jsx:

function App() {
  // ...
  const changeBackgroundColor = () => {
    apiRef.current.setBackground({
      color: [Math.random(), Math.random(), Math.random(), 1],
    });
  };

  return (
    <div className="App">
      <button onClick={changeBackgroundColor}>Change background color</button>
      <Viewer apiRef={apiRef} />
    </div>
  );
}

Save, click on the button, voilà!

If it doesn't work, check that you have clicked on the play button of the viewer beforehand to load the model ;)

Change the chair color

Alright, that function is a bit bigger but you'll have to trust me.

function App() {
  // ...

  const changeChairColor = () => {
    apiRef.current.getMaterialList((err, materials) => {
      const plasticMaterial = materials.find(
        (material) => material.name === "Blue plastic"
      );
      plasticMaterial.channels.AlbedoPBR.color = [
        Math.random(),
        Math.random(),
        Math.random(),
      ];
      apiRef.current.setMaterial(plasticMaterial, () => {
        console.log("Updated chair color");
      });
    });
  };

  return (
    <div className="App">
      <button onClick={changeBackgroundColor}>Change background color</button>
      <button onClick={changeChairColor}>Change chair color</button>
      <Viewer apiRef={apiRef} />
    </div>
  );
}

A new color for our chair!

Hooks refactor

To abstract the initialization code and clean up our component, we'll write a custom hook that leverages all the code previously written plus a useState for the api object.

Custom hooks are one of the most powerful tools to keep our code well organized. Once you start using them, there's no going back.

import React, { useEffect, useRef, useState } from "react";

// Our wonderful chair model
const MODEL_UID = "c632823b6c204797bd9b95dbd9f53a06";

const useSketchfabViewer = () => {
  // This ref will contain the actual iframe object
  const viewerIframeRef = useRef(null);
  const [api, setApi] = useState();

  const ViewerIframe = (
    <iframe
      // We feed the ref to the iframe component to get the underlying DOM object
      ref={viewerIframeRef}
      title="sketchfab-viewer"
      style={{ height: 400, width: 600 }}
    />
  );

  useEffect(
    () => {
      // Initialize the viewer
      let client = new window.Sketchfab(viewerIframeRef.current);
      client.init(MODEL_UID, {
        success: setApi,
        error: () => {
          console.log("Viewer error");
        },
      });
    },
    // We only want to initialize the viewer on first load
    // so we don't add any dependencies to useEffect
    []
  );

  return [ViewerIframe, api];
};

export const Viewer = ({ apiRef }) => {
  const [ViewerIframe, api] = useSketchfabViewer();

  apiRef.current = api;

  return ViewerIframe;
};

What we learned

We can use refs to feed DOM elements to libraries that need them.
We can use refs to give back objects to parents. Another solution could have been to use a React Context.
We can use custom hooks to make our code cleaner and more readable.

You can find the final source code here.

Shout-out to Sketchfab for their amazing product and in particular to Arthur Jamain for the great collaboration we had over the last few months.

Can I Use Web Components

Fri, 10 Apr 2020 00:00:00 GMT

The goal of this article is to show you what is a web component, why you can now use it in production, and when you should do so.

For all those who already have some knowledge in web components, I would like to specify that I will be talking about the web component V1 standards as the v0 draft APIs are now removed from Chrome since Chrome 80.

If you’re new to web components, don't worry you can catch up starting from the v1 standards, looking at the v0 specifications and standards might not be useful unless you want to understand the journey of the web components to reach its V1.

But first, you might want to be able to understand the basic concepts of a web component.

Definition of a web component

So first and foremost, what is a web component?

"Web Components is a suite of different technologies allowing you to create reusable custom elements — with their functionality encapsulated away from the rest of your code — and utilize them in your web apps." - MDN

Simply put, the web component is not just a thing by itself. For a component to be called so, different technologies have to be combined.

What are those "different technologies" then?

I will go through the different technologies that make a web component. This article won't be able to encompass all the possibilities of a web component and all its possible implementations. For those who would like to go further into the explanations, I will provide some resources as you read through.

Custom elements

A web component is nothing more than an enhanced custom element. A custom element is a javascript class expanding the HTMLElement class that allows you to create a new HTML tag for your component.

First, you have to create a class as shown in the script below.

class MyComponent extends HTMLElement {
  constructor() {
      super();
      ...
    }
}

We then define a custom element with the previously created class.

customElements.define("my-component", MyComponent);

Your custom element is ready to be used as a tag in any web project.

<my-component></my-component>

let's try it in a simple html document:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>my first custom element</title>
  </head>
  <body>
    <my-component></my-component>
    <script>
      class MyComponent extends HTMLElement {
        constructor() {
          super();
        }
      }
      customElements.define("my-component", MyComponent);
    </script>
  </body>
</html>

And that's it! you have created a custom element. Now if you inspect your HTML document on a web browser you should be able to see something like this:

The main idea of custom elements is that you can create, extend, and reuse them everywhere in your code.

If you are looking for a more in-depth documentation, check the official documentation. I would also recommend this article on customElements that is giving you a good overview of the matter.

Shadow DOM

The next technology that we will be discussing is the Shadow DOM. When you attach a shadow DOM to your custom element you then scope all the CSS, and the HTML in this DOM. The big idea of the shadow DOM is to be able to create templates for your web components that won't be affected by any style or js selectors and therefore keep your code and feature as intended.

Let's try it in our previously created component :

First create a shadow root:

const shadowRoot = this.attachShadow({ mode: "open" });

If you want to know more about the attachShadow mode argument I strongly recommend this article from Leon Revill.

Create some elements:

const spanElement = document.createElement("span");
spanElement.textContent = "Hello world";

Create a style element:

const style = document.createElement("style");

Create a css class for your span element and add some css rules:

style.textContent = ".my-class {color: red}";
spanElement.setAttribute("class", "my-class");

And finally attach them to your shadow root:

shadowRoot.appendChild(style);
shadowRoot.appendChild(spanElement);

so here it what your component should look like now:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>my first custom element</title>
  </head>
  <body>
    <my-component></my-component>
    <script>
      class MyComponent extends HTMLElement {
        constructor() {
          super();
          const shadowRoot = this.attachShadow({ mode: "open" });

          const spanElement = document.createElement("span");
          spanElement.textContent = "Hello world";
          spanElement.setAttribute("class", "my-class");

          const style = document.createElement("style");
          style.textContent = ".my-class {color: red;}";

          shadowRoot.appendChild(style);
          shadowRoot.appendChild(spanElement);
        }
      }
      customElements.define("my-component", MyComponent);
    </script>
  </body>
</html>

Now you can try going to the console and access and modify the elements in the shadow DOM but you won't be able to.

As well if you add some style in your HTML file, it won't affect the shadow DOM elements. Try adding style on the span element for example:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>my first custom element</title>
    <style>
      span {
        color: blue;
      }
    </style>
  </head>
  <body>
    <span>Hey! I am just here for the example tho...</span>
    <my-component></my-component>
    <script>
      {...}
    </script>
  </body>
</html>

As you can see the span inside the shadow DOM is not affected by the style.

HTML Template

"The template element is used to declare fragments of HTML that can be cloned and inserted in the document by script. In a rendering, the template element represents nothing. The template contents of a template element are not children of the element itself." - html.spec.whatwg.org

This means that you can define a template for your Web component and clone it for every instance of your component.

So first you create your template:

const template = document.createElement("template");
template.innerHTML = `
  <style>
    span { color: red; }
  </style>
  <span>Hey There! I am in the template </p>
`;

Then you can clone it's content into your shadow tree:

shadowRoot.appendChild(template.content.cloneNode(true));

As I am sure you noticed, we don't just add the template in the shadowRoot but instead we copy its content. That's because Templates are not rendered so what we need for our component is the content of the template.

Let's implement it in our component:

export class MyComponent extends HTMLElement {
  constructor() {
    super();
    const shadowRoot = this.attachShadow({ mode: "open" });

    const template = document.createElement("template");
    template.innerHTML = `
    <style>
        span { color: red; }
    </style>
    <span>Hey There! I am in the template </span>`;
    shadowRoot.appendChild(template.content.cloneNode(true));
  }
}

The important point to keep in mind about templates is that it allows you to declare chunks of HTML in your document that will be rendered and that sole purpose is to be cloned and reused. It is a perfect fit for web components.

You could define it in your index.html since template elements are not rendered in the browser but I would recommend keeping it close to your component declaration to keep your index.html file clean and organize your templates by component.

You can even use the slot Element to make your template more flexible. Those of you who already worked with Vue.js may be familiar with the concept of slot. For the others Slots are like placeholder. They allow the developper to create specific elements that will be placed inside the parents component DOM at predifined emplacement.

Imagine that your component template is a text with holes, each hole has an id. When you add children to your component, you can assign them a slot attribute. This slot attribute allows your component to place them in the "holes " according to their ids (the slot attributes).

If you want to learn more about slot element with web component I would recommend this documentation from MDN.

The addition of those four standards-based technologies is what defines the web component today.

ES Modules

This part is not always explicit in the definition of a web component. It doesn't affect the component on itself but more the way you will be able to use it in a project.

"The ES Modules specification defines the inclusion and reuse of JS documents in a standards-based, modular, performant way." - webcomponents.org

The ES modules should allow developers to use the script tag with the type module. Wich means you can export variables or components from your .mjs files and import it in your .html file within a module script tag.

The ES modules standard allow developers to reuse web component in any HTML file by importing them from an mjs file.

// wc.mjs
export class MyComponent extends HTMLElement {
  constructor() {
    super();
    const shadowRoot = this.attachShadow({ mode: "open" });

    const spanElement = document.createElement("span");
    spanElement.textContent = "Hello world";
    spanElement.setAttribute("class", "my-class");

    const style = document.createElement("style");
    style.textContent = ".my-class {color: red;}";

    shadowRoot.appendChild(style);
    shadowRoot.appendChild(spanElement);
  }
}

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>my first custom element</title>
    <style>
      span {
        color: blue;
      }
    </style>
  </head>
  <body>
    <span>Hey! i am just here for the example tho...</span>
    <my-component></my-component>
    <script type="module">
      import { MyComponent } from "./wc.mjs";
      customElements.define("my-component", MyComponent);
    </script>
  </body>
</html>

An MJS file is a source code file containing an ES Module.

You will need a server to be able to fetch with import, as it doesn’t work on the file:// protocol. You can use npx serve to start up a server in the current directory for testing locally.

If you are interested in how the ES module works I strongly advise that you read this article from Lin Clark.

Can I use it?

Now, a lot of developers are still quite reluctant to use web components in their projects. And you can understand why! The lack of common standards implemented by all major browsers since it's creation in the early 2010s has allowed many web component libraries to raise and grow sub-communities that do not share the same opinion on the definition of a web component and its implementation.

On the article Why the React community is missing the point about Web Components by Ben Halpern in November 2018, Dan Abramov comments the following :

[...] The problem is that there's no single "web component community". There are multiple sub-communities. You mention "web component libraries" in the post. These libraries don't agree on a single standard so React would need to pick a side in debates like "how does server rendering work with web components". Whatever we pick will have large downstream effects, and our reluctance to support more has to do with being careful [...] about the semantics — not somehow being at odds with web components per se.

So I think some important points have to be clarified for you to understand the implication of using it in your projects.

Web components are tools

It is important to understand that Web components are not designed to be better than HTML elements. We can consider web components as a tool to extend HTML, not to replace it.

And as we know the same tool cannot be used for all projects. So before adding this tool to your toolbox make sure you really need it or else it might some unecessary weight.

Here are some points that you should consider before jumping in :

It is not that simple to write a web component. It can be quite complex and verbose depending on the scope of your web components. And most of the project with quite a large number of web components have been using some libraries to creates them. webcomponents.org has a page dedicated to those libraries.
Comparing a web component and a react component for example is like comparing apples and oranges. The definition of a web component is not setting limits to what you can put in it. React in its documentation explain it this way:

"React and Web Components are built to solve different problems. Web Components provide strong encapsulation for reusable components, while React provides a declarative library that keeps the DOM in sync with your data. The two goals are complementary. As a developer, you are free to use React in your Web Components or to use Web Components in React, or both."

Understanding and defining what will be the interfaces between web components and the rest of your project's tools is important.

Web component v1 standards are not that old and you probably want your project to work on older browsers versions too. Simply put, you might need polyfills. I will come back to it later on.
One of the down points of the web component often cited is that developers have been struggling to use it with server-side rendering. This is mainly caused by the fact that the contents of ShadowDOM are often missing when the view is delivered to the client. One possibility is to remove the shadow dom from your component like you can see in this article. Other web component advocates found ways to make it work like in this article by Steve Belovarich or this one by Peter Goes. I think you can sum it up by saying that for now there are no standards for server-side rendering with a web component but you can always find a workaround.

Just remember to take all this point in account before jumping in.

Future proof

One of the main arguments of the pro web component community is that web components are future proof. Since it is based on web Standards, it is now supported by all modern browsers and should always be.

I can already hear your ask:

"Alright that's for the future? But what about our beloved past heroes like IE?"

Well you are right to be worried about them and that should be one of your main concerns.

Below are the previously cited technologies browser supports at the current date. If you want the latest data, click on the link below each of them.

can i use custom element v1

can i use custom shadow DOM v1

can i use javascript module script tag

can i use javascript module dynamic import

can i use HTML templates

As you can see neither Internet explorer is not supporting any of them neither the early versions of other browsers.

There are polyfills for the web component APIs. You can use the official polyfills.

With those polyfills, nothing you can now develop your future and past proof web component!

Reusability

A web component is easily reusable and is trustworthy. Its strong encapsulation provides security for its data. It also ensures that it will be used the way its designer intended to.

Should I use it?

Based on previous points, there are no specific reasons to not go for it. Still, the Web component isn't mature enough to be trusted for production by most developers. It solves problems but adds others.

For now, the best use the community can agree on for a WC-based project is a design system UI components library. Tesla for example, has crossed the bridge and chosen to make its design system with web components.

Building a sustainable design system can be a real brain teaser for companies. Big companies often use a lot of different technologies and frameworks for all their products.

They reach a point where sustainability becomes a problem in terms of design and technology. They ask themselves those questions :

Do we have to create a UI component library for each framework that we use?
How can we be sure that we don’t switch framework in the next few years?
And If we make sure we don't, aren't we missing an opportunity?

A web component is not based on any framework owned by any company but on standards adopted by all major browsers. You don't have to worry about the hype effect on your chosen technology or framework. This is why web components fit if you decide to create a UI component Library or even a design system.

Of course, you can also use it in any static pages project but I wanted to highlight the one big use where you should consider it.

Conclusion

Now that the standards that define Web components are implemented in every modern browser, they are meant to be part of the future of the web.

As a web developer, it’s time to consider web components as a practical solution for your features and problems.

I would like to quote Serhii Kulykov(@webpadawan) on this article

"Now in 2019, it’s time to give a try to Custom Elements and Shadow DOM. They might not be “the next big thing” by themselves, but rather solid foundation the next big thing could be built upon." - Serhii Kulykov

Of course when the technology will be more mature, accepted and integrated into our projects we will witness the apparition of new ideas and maybe even new standards.

But for now, we are the ones who have to be bold enough to try on our own.

If you are interested in the subject, I recommend the following articles:

The journey of Web Components: wrong ways, lacking parts and promising paths - Serhii Kulykov

Why I don't use web components? - Rich Harris (svelte creator).

I recommend you read the article but most importantly to read its comment. Especially this gist that is a direct answer to the previously mentioned article

Why the React community is missing the point about Web Components - Ben Halpern.

Once again the interesting part is mostly in the comments with a discussion between Dan Abramov et Benny Powers

Optimizing for high availability in time of crisis

Thu, 09 Apr 2020 00:00:00 GMT

These past few weeks have been tough for many websites. Some of them find themselves with an unprecedented increase in traffic (e.g. video conference software, online grocery stores, social media) and others with a dangerous reduction of users (e.g. car insurance, travel agencies, etc).

Although both of those scenarios present severe risks for businesses, the second one often cannot be solved by technology alone. So the goal of this article is to help with the former and provide a set of quick actions to maintain a highly available website on AWS under unexpectedly high traffic.

Most of the time in this situation, getting our website back online as soon as possible will be our top priority, which doesn't leave much time for writing or changing code (and would we want to add untested code into the situation anyway?). It also means that we're prepared to spend more on our infrastructure to accommodate the increased traffic (which may well pay for itself if that helps generate more business). For each recommendation I'll try to give a sense of the price increase involved.

Some of the actions below are pretty generic (only assume a standard 2- or 3-tier web infrastructure) and some are more context dependent. We'll cover the generic actions first and then explain in which context the specific actions apply.

Generic actions

1. Increase instance size

Time: < 1h

Money: from ×2 to ×100 on EC2 bill (you choose)

Effectiveness: increased capacity for cpu- or network-bound websites

This is the simplest and quickest action, although it can be a little bit expensive depending on the new instance type we choose.

Keep in mind that increasing instance size does require replacing all the currently running ec2 instances (we can't increase the size of an existing instance), so this migration requires some planning. In particular, if our application is stateful, we risk losing data so our best option is probably to take snapshots of EBS volumes and recreate them for the new instances. In most cases however all that's needed is a redeployment of the app on the new instance(s).

To save on ec2 costs remember to use reserved instances when possible.

2. Use autoscaling

Time: 2h - 6h

Money: from ×0.1 to ×10 on EC2 bill depending on autoscaling rules

Effectiveness: lifts all cpu- or network-bound blockers for stateless apps

This is a very good option if our app is already stateless.

Autoscaling will automate the creation and termination of EC2 instances to match the fleet size to the current traffic. This will allow us to not over-provision instances to serve our users and save a lot of money. Remember that we can set a max number of instances in our fleet so we can stay in control of billing.

If our app isn't stateless already, we may want to consider focusing development efforts to make it so in the short term. Although it's likely to require substantial change to the code, it's also going to unlock big performance improvements through parallelization (and autoscaling) so it might be worth the trade-off. If however we decide that making our application stateless isn't for us, there still are still things we can do to unlock performance, especially option 6.

3. Deploy to multiple regions (active/active sites)

Time: 0.5 day (with IAC, prepared) to 3 days (without IAC, unprepared)

Money: ×2 for the entire AWS bill

Effectiveness: doubled capacity

Another way to keep response times low worldwide and to massively increase our website's resilience is to deploy it in multiple regions, both serving traffic (also called active/active deployment). By this point we should already be using multiple availability zones in the first region and, by deploying a copy of the site to a second region, we will get:

the capacity to serve double the normal number of users ;
a site that's resilient against region-wide disasters.

Infrastructure as code (or IAC, with e.g. Terraform or CloudFormation) will make this deployment much easier and faster. If we don't use such a solution already, I wouldn't recommend setting one up just now, as it usually takes longer to prepare an IAC script than to provision an environment manually through the console. However we should definitely add this to our backlog for when things start to calm down.

Once we've got our second region running, we'll need to split the traffic between the sites. This can be done with Route53 which has several routing policy types to handle this, including latency-based or geolocation routing.

Finally we probably need to use a "single" database for both regions. One way to achieve this is to create a read-replica of the existing RDS database in the second region (with async replication). Point all database access to the master database in the first region and in case of disaster in zone 1, we can promote the replica (AWS replaces the dns automatically, no code or configuration changes required).

Specific actions

4. Cache static assets

Time: 0.5 to 1 day

Money: depends on asset sizes and volume downloaded, see calculator

Effectiveness: removes static files load on web servers

This is useful if web servers take time to serve each and every request for static assets (e.g. images, CSS and JS scripts, static html) or, even worse, re-generate them each time. In that case a great way to reduce the load on the servers is to have AWS serve the static assets for us.

A standard solution to this problem is to store assets in S3 and serve them through AWS's CDN CloudFront. Using CloudFront enables caching of resources and uses edge locations around the world to ensure fast response everywhere, however it becomes expensive with many users. Use AWS's simple monthly calculator (the newer AWS pricing calculator doesn't support CloudFront yet) to estimate how much it would cost.

5. Cache long-running database queries and computed value

Time: 1h to 1 day depending on web framework

Money: 15 USD/month (with instance reservation) to 20k+ USD/month depending on requirements

Effectiveness: removes expensive computations from web servers

It's an almost unavoidable issue on dynamic websites with growing data: unoptimized database queries start taking longer, eventually blocking worker processes for several seconds at a time and seriously limit our ability to scale the number of concurrent users. While it's important to optimize database queries as much as possible, there's a simple way around this issue: caching query results.

A typical way to implement application-level caching on AWS is ElastiCache. We'll pick between a Redis and a Memcached backend based on what's easiest to implement in our application code and use the cached value as often as is reasonable.

The same reasoning (and solution) applies for computed value that are not user-dependent and don't need to be recomputed on each request.

6. Store sessions data properly

Time: 1h to 1 day depending on web framework

Money: 10 USD/month (with instance reservation) to 1k+ USD/month depending on number of users

Effectiveness: removes lock acquiring bottleneck

In case our current application has to remain stateless and we're storing session information on the servers, make sure to store the session data in a way that allows concurrent access to the session store.

For example, in some implementations (especially in PHP), storing sessions in a single file can lead workers processes to waste a long time acquiring the session file lock on every request. An alternative to that is to store session data in memory, for example using ElasticCache on AWS (see action 5).

7. Rewrite bottleneck services

Time: depends on service

Money: however much dev time worth of money

Effectiveness: removed application level bottleneck

This one is for last resort, when no other options are obviously available. Once we've identified (one of) the subsystems that are responsible for performance bottleneck(s) (for example using AWS X-Ray, New Relic or plain logs), it's time to consider whether a rewrite would yield significant improvements.

In this sort of situation it's beneficial to start relying on cloud services to be able to build quickly and cleanly. AWS services like SQS (for decoupling), AWS Chatbot, Elastic Transcoder, Lambda, Amazon Polly and Kinesis for example are reliable building blocks to build solid, serverless workflows for chatbots, video transcoding, natural language processing or event streaming quickly. And there are many similar services we can use to build extremely scalable applications.

Need more help? I'd like to help you with your scalability issues! Contact me by email or on twitter and we'll try to find a solution that's right for you. 🙂

How DUOLAB uses artificial intelligence to improve your skin

Fri, 03 Apr 2020 00:00:00 GMT

For a long time people (including myself) have used the same skin care routine everyday. Only occasionally do we switch up our products for a different brand. The creams are opened and resealed multiple times a week, resulting in the products containing preservatives and other unnatural ingredients to prevent them spoiling. Innovative new start-up Duolab aims to improve the way we do skincare, their slogan being

‘Your life is no routine. Why would your skin care be?’

In November 2019 I started working for Theodo UK, a London-based software development agency which delivers bespoke web and mobile solutions for both start-ups and world-renowned clients alike. I was very excited to hear that my first project would be working within a scrum team to deliver DUOLAB’s web and mobile application to accompany the launch of their cutting-edge skincare product. Having a strong interest in skincare myself, I was satisfied with the project I was about to embark on!

Everybody’s online presence and engagement is more important than ever given the current Coronavirus pandemic and home lockdown! DUOLAB is ahead of the curve by creating a future proof product to personalise your skin care without the need of a dermatologist or a shop assistant to provide a recommendation. You’re now able to improve your skin from the comfort of your own home.

DUOLAB’s artificial intelligence (AI)-driven personalised skin care is revolutionising the way we care for our skin. Unlike traditional skincare treatments, DUOLAB utilises an AI skin diagnostic tool within its mobile app to calculate a tailored, personalised skincare protocol.

The DUOLAB device blends a suitable combination of a moisturising base and a targeted concentrate which has been recommended to the customer by the skin diagnostic tool. The skin diagnostic process begins with a guided selfie which is inputted into the AI model. The AI assesses various skin traits, such as blemishes and wrinkles, which gives rise to more accurate recommendations than ever before. The selfie is followed by a short quiz which further determines the base-concentrate combination.

Your skin changes over time, so customers can retake the skin diagnostic when they choose - this will give them an up to date recommendation.

Theodo UK’s comprehensive technology stack allows us to approach a broad range of clients with solutions to their business needs and problems. In the development of the DUOLAB mobile app, we used Facebook’s React Native framework to create a cross-platform, native app with an excellent user experience. Our use of Amazon web services (AWS) for hosting the mobile app and website was crucial in getting the project set up and running quickly. AWS gave us the reliability and flexibility that we needed to build our products with confidence.

Our best-in-class products and quick turnaround time are testament to our position in the global start-up studio, M33. Theodo’s M33 association enables us to leverage our relationships with other M33 companies to diversify our final product offerings. During the DUOLAB project, we worked alongside fellow M33 companies, Sicara and BAM Mobile Limited, who were responsible for the skin AI analysis and mobile app designs respectively. The shared philosophies of the seven M33 companies has enabled them to develop a robust, synergistic relationship which is capable of tackling the digital needs of modern companies.

In January 2020, DUOLAB introduced their skincare solution to the world at CES® 2020 in Las Vegas - ‘The global stage for innovation’. The mobile app was demonstrated to consumers before its launch and it was well received.

Since then DUOLAB has launched the product, mobile app and website within the UK. The products are on sale in selected L’Occitane stores and on the DUOLAB website. Given the current situation, DUOLAB has adapted to the needs of the consumer in a time when many other competitors haven’t. I hope that you can take these lessons from DUOLAB and stay ahead of the curve.

Serverless: a New Paradigm

Fri, 03 Apr 2020 00:00:00 GMT

Since its creation in 2014, AWS' Serverless has delivered by reducing the cost of infrastructure management while increasing speed, agility, and scalability.

What is Serverless?

“Serverless allows you to build and run applications and services without thinking about servers” — AWS

Serverless shifts more responsibility to Cloud Providers, allowing IT teams to build applications and services without managing the underlying architecture. Developers can send code, run databases, store data and manage security, without thinking in terms of servers, networks and patching. One classic example of a Serverless service is FaaS (Function as a Service), a Serverless approach to Compute that allows developers to send application code to their Cloud Provider and have it run without dealing with the underlying complexities of the server or network or other infrastructure components.

Today the three main Cloud Providers for Serverless services are Amazon Web Services, Microsoft Azure, and Google Cloud.

“A Serverless solution is one that costs you nothing to run if nobody is using it.” — Paul Johnston, ServerlessDays CoFounder

Prerequisites for Choosing Serverless in Your Organization

Splitting Your Business Logic Into Events

The first step in architecting an event-driven system is to understand the business domain through its events. Domain-Driven-Design (DDD) is a great framework to achieve this and Event Storming workshops can help business and technical stakeholders understand the processes and systems at play in a given domain.

These events are represented in business terms, e.g. Order Placed. But they ultimately trigger different Serverless services, for instance, running code, writing data, and authentication.

Code Compatibility

Because the feature is hosted and deployed on a Serverless Cloud Provider, it must follow its rules.

As you shift more responsibility to your Cloud Provider, there are some constraints around the code you write. Primarily, the language teams use has to be compatible with their chosen Cloud Provider. For example, Google Cloud only accepts functions written in Go, Node.js, and Python. However, custom runtimes make any language possible.

Serverless Pros and Cons

Pros:

Automatic scaling

The Cloud Provider ensures that the function responds consistently every time it is called. For example, iRobot faces a huge peak at Christmas when new robot owners start their device, AWS will automatically scale compute power to absorb the load.

In a Serverless paradigm, the Cloud Provider manages the running of your services, eliminating manual provisioning.

You only pay for what you use

Serverless Cloud Providers charge based on resource usage only as opposed to paying for allocated servers that most of the time are only used at a fraction of their capacity. No need to pay constantly for the extra buffer of compute power you might need during peak periods. And even better, if your function isn't used, you don't pay.

When you consider that organizations use only a portion of the actual computational power of their servers, you realize the potential gain is significant. However, these servers that sit unused still consume energy, which leaves room for improvement to make the cloud computing industry greener.

Reduce the Ops team size needed to run your systems, reducing your TCO (total cost of ownership).

As there is less infrastructure managed by the team, they can focus on delivering value to users, rather than running the underlying systems.

Source: Serverless.com

Cons:

Vendor Lock-In

As you become serverless, your systems couple to the underlying Cloud Provider. This makes it difficult to move to another Cloud Provider later on, for instance, if their pricing changes. Certain large organizations are concerned by this dependency and therefore want to remain cloud agnostic. There are some solutions to reducing the extent of vendor lock-in, but these often come at the cost of some original benefits of Serverless.

Requires a mindset shift (a new way of coding)

Serverless can be massively empowering to developers, but it requires a mindset shift, correct coding, and consistent approaches to security and compliance. In this video, Nicole Yip, Senior Infrastructure Engineer at Lego, explains how their journey to Serverless was tightly linked to instilling the DevOps culture within the team.

Changes certain security practices

Serverless removes certain security concerns, like patching operating systems as it is handled by the cloud provider who seamlessly applies security patches between two executions of a serverless function. Other security practices remain unchanged, like using a Web Application Firewall (WAF). But serverless also introduces an unprecedented level of flexibility in the definition of privileges at the function level. For instance you can specify that a given function should only be able to query certain elements of your database. This flexibility can prove itself dangerous as a rushed developer might be tempted to set lax security rules. This is why serverless requires adequate coaching and automation to ensure those rules are properly set.

Find out more on security with our expert Ben Ellerby’s talk at ServerlessDays.

Cautions

If your code is used at high intensity, the Serverless solution might cost more for each query than the marginal cost your organization pays for conventional hosting.

Parts of your application that would not benefit from permanently allocated resources can still be outsourced to a Serverless function. Authentication is a classic example where you would use a serverless authentication service like Cognito.

Use Cases

Serverless architectures have flourished. And use cases have grown in a broad range of industries and companies of all sizes such as:

Netflix: Uses AWS for automatically encoding media files, automation of backup management, monitoring of other AWS services.
PayPal: Has moved mission-critical workload to Google Cloud and transformed the way its 5,000 developers work.
CodePen: A single SRE (Site Reliability Engineer) manages a Serverless infrastructure capable of responding to more than 200,000 requests in times of high demand, reducing the cost of managing infrastructure.
Lego: After a system failure during Black Friday, the team decided to move to Serverless to handle peaks for their online shop. Their team now quietly watches their systems auto-scale.

In-House Serverless (An Alternative to Cloud Providers)

Since 2016, various frameworks have been gaining momentum, including OpenFaas and Knative. These solutions make it possible to no longer depend on Cloud Providers like AWS or Azure and allow you to replicate the code execution aspect of Serverless applications based on Kubernetes to host them yourself. However, your organization is again responsible for the servers on which your code is run.

Large corporations usually have large IT departments, maybe a few data centers and often an intricate plan to migrate to the cloud. Such companies could benefit from the speed and agility provided by Serverless Compute (Functions as a Service or FaaS) that would be enabled deploying Knative on their own infrastructure. The complexity of managing would not be outsourced to a Cloud Provider but handled by the existing IT team but the development team would highly benefit from the possibilities that Serverless has to offer.

Conclusion

Serverless is recognized for bringing more agility and scalability to the organization. In addition, Serverless can help large organizations reduce the time-to-market of new products.

By breaking up a monolithic architecture in microservices, developers can focus on business-specific code and deliver customer-centric solutions. Where appropriate, it is an excellent solution to reduce the cost of managing the infrastructure and increasing the impact of your developers on your business.

Whether you want to start a new project with state-of-the-art technology or are looking to reduce the cost of managing your infrastructure, Serverless is a technology to seriously consider.

Ensure code quality; create your own ESLint rules

Wed, 01 Apr 2020 00:00:00 GMT

Ensure code quality; create your own ESLint rules

Introduction

We used ESLint rules to enhance code quality. This article is about how one would proceed to create one's own custom rule. We used this procedure at Theodo to make sure our frontend code was XSS free on every project.

If you are not familiar with XSS vulnerabilities here is an article from OWASP that summarizes well what it is about.

Context and Needs

There are over 80 developers at Theodo, working on over 30 different projects. This setup presents some challenges that many tech companies face:

Code consistency and maintainability.
Continuous improvement of the code quality
Optimization of the development environment
A straightforward onboarding for the newcomers on the different projects

We came up with a solution that is easy to implement and extremely effective: creating a custom linting rule for ESLint.

I will use an example of a rule we needed for our frontend projects with an objective of no know XSS vulnerability in our code.

We started by implementing the rule to React and Vue, which covers the majority of our projects. There are a few differences to take into account when developing a rule for each of these frameworks. We will look at them in this article.

If you encounter issues configuring ESLint, especially alongside other fromatting tools such as Prettier, I recommend this article that tremendously helped me to configure my development environment.

Strategy

Nobody is infallible. Creating an automated check that alerts the developer is more reliable than documentations on the project or code reviews. Don't get me wrong, code reviews are essential, as is a well-written documentation, but it is often not enough to enforce a coding practice. Having an automated rule also means shorter code reviews and an easier time following the best practices.

Implementing a rule that will be in the IDE also allows you to suggest the best way to do something, as soon as it is written. This is a roundabout way to train your developers, that isn't time-consuming for you.

How to implement an ESLint rule

To understand how a rule is coded, we need to take a look at our code's Abstract Syntax Tree and how we can use it.

The Abstract Syntax Tree (AST)

An AST is a graph representation of your code. ESLint uses it to navigate your code and to decide which actions to take and where.

The screenshot below is an example of a simple Hello World! AST of a React script, containing a dangerouslySetInnerHtml. We can see the tree structure and the names of the different nodes. For example, the FunctionDeclaration node contains the information from the function keyword, up to the closing tag.

The screenshot below is an example of a simple Hello World! AST. We can see the tree structure and the different nodes' names. For example, the FunctionDeclaration node contains the information from the function keyword, up to the closing tag.

This screenshot is taken from astexplorer.net which is an extremely useful resource to visualize and navigate your AST.

It works nicely for Javascript and React. However, when exploring your Vue AST, you will need to find another way for the <template> part of a file, which is parsed as HTML. AST explorer's own HTML configuration does not match perfectly with what my Vue code outputs. I used this tool instead, which is the same, except only for Vue.js

We'll come back to the particularities of Vue.js later in this article.

How ESLint uses your code's AST

ESLint uses a visitor that will parse your AST for each file it is allowed in. In this visitor function, you can define actions for each node type. If we use the same example as above, we could write something like this.

    const create = context => {
      return {
        Program(node) {
          try {
            console.log('Entering the program!');
          } catch (error) {
            context.report(node, error.stack);
          }
        },
        FunctionDeclaration(node) {
          try {
    				console.log(`Function ${node.id.name} is defined`);
    			} catch (error) {
    				context.report(node, error.stack);
    			}
    		}
    	}

In the example, I use the root node Program to trigger some action, and the FunctionDeclaration node.

This is a very basic example, where we can already see a few important things about the way ESLint will work.

The context object

The context passed as an argument here contains information relevant to the rule. We won't use it here, but if you want more info here's the official documentation: https://eslint.org/docs/developer-guide/working-with-rules
ESLint uses the node names to trigger some actions. Here, a console.log. These actions will be triggered for each node with the same name in the AST. For example, there is only one Program, but usually more than one function declaration. In the above example, our console output will look like:
```
Entering the program!
Function example is defined
Function helloWorld is defined
```
Each node has access to its sub-nodes, and also to its parent node if it has one. This is why we can use node.id.name to get the name of the declared function. The Identifiernode 'id' is a sub-node of the FunctionDeclarationnode, just like the BlockStatement node named 'body'.
Our approach to implementing our custom rule

Test-Driven Development works like a charm in this particular case. We obviously want to test our rule, but there are other advantages to doing so.

For those who are not familiar with Test Driven Development (TDD), it consists of three phases.

Write a test case you want your code to pass. Run the tests. It fails. (Red test)
Write code to pass your test, not more. Run the tests, it should pass. (Green test)
Refactor your code.

This process is often shorthanded in Red-Green-Refactor.

The TDD method is awesome and you should use it where it fits. It does fit here for a few reasons.

One of them is that it allows us to ship a stable and well-documented version, with examples based on the tests.
In the context of our project, we were able to quickly test our custom ESLint rules in real-life projects, and obtain feedback. It was easier to convert the feedback in tests first (red tests), and then write some code to fix the issue (green tests).
In our case, many of our tests were easy to write, since they often were chunks of code we wanted to be flagged, or not. However, turning them green is less trivial. Here's an example:

    // Test cases we do not want to catch with our rule
    testCase(`
          <template>
            <div class="content">
              <div v-html="message" />
            </div>
          </template>
    
          <script>
            import DOMPurify from 'dompurify';
            const rawHtmlInput = '<a onmouseover=\"alert(document.cookie)\">Hover me!</a>';
            export default {
              name: 'HelloWorld',
              data () {
                return {
                  message: DOMPurify.sanitize(rawHtmlInput)
                }
              }
            }
          </script>
        `),
    
    // Test cases we do want to catch
    testCase(`
          <template>
            <div class="content">
              <div v-html="message" />
            </div>
          </template>
    
          <script>
            import DOMPurify from 'dompurify';
            const rawHtmlInput = '<a onmouseover=\"alert(document.cookie)\">Hover me!</a>';
            export default {
              name: 'HelloWorld',
              data () {
                return {
                  message: rawHtmlInput
                }
              }
            }
          </script>
        `),

These two cases are similar, except in the message part, one is sanitized by DOMPurify whereas the other isn't.

And more generally, tests protect us from breaking previously working code, and catch some edge cases during the development phase.

Managing the Vue AST

As teased above, let's look at the particularities of Vue files when writing an ESLint rule.

Vue files are split in 3 parts, a <template>part, a <script>part and a <style>.

The AST, in this case, does not include a unified version of the file. It is split in the same way. This means our visitor function will have to be split as well.

I use a standard ESLint parser for Vue, that exposes a function which takes the context and 2 visitors as arguments. One for the template, and one for the script.

Here is a code snippet to give you a better idea:

    const create = context => {
      // The script visitor is called first. Then the template visitor
      return utils.defineTemplateBodyVisitor(
        context,
        // Event handlers for <template>
        {
          VAttribute(node) {
            try {
              // SOME CODE HERE
            } catch (error) {
              context.report(node, `${utils.ERROR_MESSAGE} \n ${error.stack}`);
            }
          },
        },
        // Event handlers for <script> or scripts
        {
          Program(node) {
            try {
              // SOME OTHER CODE THERE
            } catch (error) {
              context.report(node, `${utils.ERROR_MESSAGE} \n ${error.stack}`);
            }
    			}
        },
      );
    };

From there it is basically the same approach as a single visitor function, like with vanilla Javascript, duplicated in each function.

I used the library eslint-vuejs (https://eslint.vuejs.org/developer-guide/#working-with-rules) that already provides a function to manage the template and script part at the same time :

context.parserServices.defineTemplateBodyVisitor(visitor, scriptVisitor) where both visitor and scriptVisitor are functions like the one presented before.

And there you go, you can check any number of things you want to alert your user about.

In our Vue example, we checked for v-html in the template and checked if it was verified with libraries like DOMPurify. We know v-html is generally not the way to go, but sometimes it is a necessity, and we wanted to make sure we weren't introducing a potential XSS.

Here is what the custom rule looks like in action:

If you also think that integrating a custom ESLint rule to produce better code is awesome, definitely check out the project I used as an example, it open-sourced here: https://github.com/theodo/RisXSS

Conclusion

In conclusion, integrating custom ESLint rules in your projects will help improve the code quality, the lead time of your features, since fewer mistakes will make it to the code review part of the process, and the developer's experience.

With a rule matching a bad practice, you can easily suggest a good practice right in the IDE, and the developers will quickly and painlessly learn it.

In our case, after some pilots, the rule was used in the development environment of the developers and guarantees us a certain level of code security.

If you want to dive deeper into the custom rules world of ESLint, a good place to start is the official documentation for developers: https://eslint.org/docs/developer-guide/working-with-rules

I will also link here two articles that helped me greatly with ESLint custom rules. If you found mine interesting, you should definitely check these out.

Writing Custom Lint Rules for Your Picky Developers

Custom ESLint Rules For Faster Refactoring

How real developers use a CMS to build a showcase website with Netlify, Nuxtjs, and Contentful

Tue, 24 Mar 2020 00:00:00 GMT

Why build a CMS driven showcase website?

Here you are, you just started a project to build the latest shiny showcase website for your client. Now comes the difficult part: you want to offer your client the best bang for his bucks, not wanting to redo part of the website each time the content changes. In the same way, you are probably not in charge of coming up with the shiny design but coding it, why would you lose time adjusting wordings, coming up with translations?

That's where a Content Management system (CMS) becomes handy, giving your client an interface to edit the content of his site without wasting its precious developers' time. Now you might be thinking 'A CMS, last time I gave a go-to WordPress, I wasn't convinced !' and I agree with you. Today CMS fall into 2 usages:

A what you see is what you get, that generates a website: WordPress, Drupal are some examples
headless CMS which are a back-end only content management system built as a content repository that makes content accessible via a RESTful API: WordPress, Drupal are part of the game here as well, but we will use in this tutorial a concurrent specialized for this usage: Contentful to convince you that working with a CMS can be enjoyable.

Why use a server-rendered static website for a showcase website?

Doing so leverages the advantages of using a progressive web application with server-side rendering (more about SSR here) without the cost of running a server since you just need to serve your files. Furthermore, we leverage the following advantages for a showcase website:

SEO friendly
Performance
Monthly infrastructure cost

Why use Netlify & Nuxtjs

Netlify

Infrastructure wise our requirements are very small, we only need a CDN to distribute our site. We could use any major cloud provider directly, however, I decided to go with Netlify service for the following reasons:

The integration is exceptional: creating a CI that builds and deploys your website in production has never been easier.
Https works out of the box as long as you have your own domain.
Netlify service is built on top of AWS which means we don't have to worry about reliability.

Nuxtjs

Nuxtjs is a server-side rendering framework built on top of Vuejs. For our case, Nuxtjs allows for dead easy creation of a server-side rendered static website.

Building a CMS driven showcase website

Creating a Contentful CMS

First, as the website is content-driven, we will start by creating the CMS using Contentful to provide content to the website.

Start by creating an account on https://www.contentful.com/sign-up/ (use free trial for this tutorial) Once you sign up, you can create a new space (which is Contentful way of creating a project)

In the popup dialog set the type of space (you can upgrade later), the space name (the project name):

Space type: free
Space details: data_driven_static, empty space

You have successfully created a new contentful project

Creating the first content

Contentful is organized between:

Content Models which is where you define the structure of the content
Content which is where you can contribute (create content)
Settings where you will find some useful settings later on.

Create the content model

To create our first content we will use the Content model tab, click on add content type

When creating a new content type, we have to fill in some important information:

The Name of the content type, which is the displayed name for the content type: we will use Section.
The API Identifier is the name of the entity when querying the Contentful API. In most cases it’s better to keep the given API Identifier, to stick as close as possible to the content type name to ease identifying the content types in the code.
Lastly, I recommend filling in the description of the content type, I will use: Basic block of content for a page.

Finally, confirm to create the new content type.

Now, we have a new empty content type. To fill it, we will add some fields:

A Title for the section
A Text for a paragraph

To do so, first, click on add field:

For the type of the field, select: Text
For the settings of the Text field:
- Name: Title
- Short text
Click create and configure
Then in settings:
- Select: This field represents the Entry title
Switch to validation:
- Make it required
Finally switch to the Appearance tab:
- Leave it as a single line
- Enter a Help text: The section title
Click Save

You successfully created the first field, we will create a second one for the paragraph:

Click add field
In creating:
- Select type Text
- Name: Paragraph
- Long text
In Appearance
- Select Multiple lines
Click create and configure
Click Save

Once you created both fields, you can now hit save on the top right corner of the content type as we finished creating our content type.

Contribute the first section

As we now have the first content type, we can contribute to creating new content. To do so, in the main contentful header, head to the tab Content.

By default contentful filter with the current content type, we can create a new section hitting the button Add Section

You end up on the form to edit the entity. You need to provide a title and you can provide some text for the paragraph. Then hit publish.

You are done for the CMS part (keep it open for latter).

Installing Nuxtjs

First, we are going to create a Nuxt application.

To do so, you need to have Npm installed and type in:

npx create-nuxt-app <project-name>.

Once you launch the command, you will need to answer a few questions:

Project name: cmsdrivenstatic
Project description: /
Author: /
Package manager: / (I will use Npm)
UI framework: you can use any of them as you like (I will not use one here)
Custom Server framework: None
Nuxt.js modules: Axios, Dotenv
Linting tool: as you like, I go with and recommend all of them (ESLint, Prettier, Lint staged files, Stylelint needs some configuration hence I will not use it here)
Test framework: I will not talk about tests here, but it’s obviously recommended to use tests (both Jest and AVA are great)
Rendering mode: Universal (SSR)
Dev tools: use jsconfig.json, if using vscode as your IDE

You can run the Nuxt dev server using:

cd cms_driven_static
npm run dev

You are done installing Nuxtjs

Fetch the section content in Nuxtjs

We will now display the Section content in the Nuxt frontend. To do so, we need to install the contentful module: npm install contentful --save.

Then to use the contentful package in Nuxt, we will create a plugin. Plugins are the recommended way to use javascript libraries globally.

Finally, we will use this plugin to fetch content in the application pages.

Creating the Nuxt.js plugin

In the plugins directory of the project create a new file: contentful.js

Add the following code in the file:

// First we import the contentful node module
const contentful = require('contentful')

// Those are set via `env` property in nuxt.config.js or environment variables
const config = {
  space: process.env.NUXT_ENV_CONTENTFUL_SPACE,
  accessToken: process.env.NUXT_ENV_CONTENTFUL_ACCESS_TOKEN
}

// Create a client to setup fetching content
const client = contentful.createClient(config)

// Our first method to fetch all section content type
client.getSectionContent = () =>
  client.getEntries('', {
    content_type: 'section'
  })

export default ({ app }) => {
  // Add the function directly to the context.app object
  app.contentfulClient = client
}

Now we will register the plugin in our Nuxt config, head to the file nuxt.config.js. In this file you will find an empty plugins array. Add { src: '~/plugins/contentful' }, in this array:

-  plugins: [],
+  plugins: [{ src: '~/plugins/contentful' }],

That's all for the plugin configuration.

Finally, we need to specify the contentful credentials, to do so, we will use the dotenv module:

In nuxt.config.js add '@nuxtjs/dotenv' in the buildModules array:

  buildModules: [
    '@nuxtjs/eslint-module',
+   '@nuxtjs/dotenv'
  ],

Then in your .env file, you need to add two variables: NUXT_ENV_CONTENTFUL_SPACE and NUXT_ENV_CONTENTFUL_ACCESS_TOKEN:

Using the prefix NUXT_ENV_ enables Nuxtjs automatic injection of environment variables into process.env.

To fill in those values go back to contentful. Go to the settings tab and find ‘api keys’. If one already exists, use it, else create a new one. (specify any name and description)

Use both Space ID and Content Delivery API - access token to fill in your .env variables.

Restart your server (interrupt and npm run dev), the webpage should load errors free.

You should be set to fetch your content on your Nuxtjs page.

Display the section content

Let’s make this API call and display the section content.

Head to the file pages/index.vue. It’s the entry-point of your nuxtjs application, namely the landing page.

A brief introduction to Vuejs single file component syntax

A single file Vue.js component is organized in 3 parts:

<template>
  <p class="paragraph">{{ greetings }} World!</p>
</template>

<script>
  module.exports = {
    data() {
      return {
        greetings: 'Hello'
      }
    }
  }
</script>

<style scoped>
  .paragraph {
    font-size: 2em;
    text-align: center;
  }
</style>

The template part which is the rendered HTML for the component
The script part which contains the component logic
The style part which is, no surprise, the CSS style applied to the component.

Fetching the section content

First, we are going to fetch the section data and we want to fetch it server-side to be able to render the full HTML before sending it to the client.

To do so add an asyncData method below the component in the script HTML element, ending with the export looking like this:

export default {
  components: {
    Logo
  },
  async asyncData(context) {
    const sections = await context.app.contentfulClient.getSectionContent()
    console.log(sections.items)

    return {}
  }
}

Reload your page http://localhost:3000/, you should see your Section item in the console.

Then we will add some logic to the contentful plugin to map the interesting properties for the landing page Vuejs component:

// First we import the contentful node module
const contentful = require('contentful')

// Those are set via `env` property in nuxt.config.js or environment variables
const config = {
  space: process.env.NUXT_ENV_CONTENTFUL_SPACE,
  accessToken: process.env.NUXT_ENV_CONTENTFUL_ACCESS_TOKEN
}

// Create a client to setup fetching content
const client = contentful.createClient(config)

// Our first method to fetch all section content type
client.getSectionContent = () =>
  client
    .getEntries('', {
      content_type: 'section'
    }) // map the interesting properties from section
    .then(({ items }) => {
      return items.map((section) => {
        return {
          title: section.fields.title,
          paragraph: section.fields.paragraph
        }
      })
    })

export default ({ app }) => {
  // Add the function directly to the context.app object
  app.contentfulClient = client
}

And use it in the asyncData of pages/index.vue:

 async asyncData(context) {
    const sections = await context.app.contentfulClient.getSectionContent()
    console.log(sections)

    return {}
  }

Display the section

Finally let’s display our data, first make the sections available in the state of the component:

async asyncData(context) {
  return {
    sections: await context.app.contentfulClient.getSectionContent(),
    }
}

Update the template to use the new data from the asyncData:

<template>
  <div class="container">
    <div class="logo-container">
      <logo />
    </div>
    <div v-for="section in sections" :key="section.title" class="section">
      <h1 class="title">
        {{ section.title }}
      </h1>
      <h2 class="subtitle">
        {{ section.paragraph }}
      </h2>
    </div>
  </div>
</template>

Update the style of the component:

<style scoped>
  .container {
    margin: 0 auto;
    min-height: 100vh;
    display: flex;
    flex-direction: column;
    justify-content: center;
    align-items: center;
    text-align: center;
  }

  .logo-container {
    height: 50%;
    width: auto;
  }

  .section {
    margin-top: 20px;
  }

  .title {
    font-family: 'Quicksand', 'Source Sans Pro', -apple-system, BlinkMacSystemFont,
      'Segoe UI', Roboto, 'Helvetica Neue', Arial, sans-serif;
    display: block;
    font-weight: 300;
    font-size: 60px;
    color: #35495e;
    letter-spacing: 1px;
  }

  .subtitle {
    font-weight: 300;
    font-size: 21px;
    color: #526488;
    word-spacing: 5px;
    padding-bottom: 15px;
  }
</style>

Then reload your homepage, you will be able to see your new section content:

Congratulations

Let’s add one more section content in Contentful to check that our page works correctly.

Once you add the new section content and publish it you should see both sections on your homepage:

Deploy the website using Netlify

Setting-up automatic deployment

Finally, we will deploy the website, making it available on the Internet.

First of all, create a repository for the project on Github or Gitlab and push your project.

Once you are there, you can head to Netlify website: https://www.netlify.com/, create an account using your favorite authentication: https://app.netlify.com/signup

Once on the project page, I recommend using ‘New site from Git’ to setup up a static website deployment with the added benefit of an out of the box working CI, building your site with every new commit on your production branch.

Select your repository: If it doesn't appear in the list click on Configure the Netlify app on Github

For your site settings, go for:

Base directory: not set -> useful if you use a mono repository to set the directory where you can build the front
Build command: npm run generate
Public directory: dist
Click show advanced to add environment variables matching the ones defined in .env:
- NUXT_ENV_CONTENTFUL_SPACE
- NUXT_ENV_CONTENTFUL_ACCESS_TOKEN

Your site is now online!

You can find the Netlify generated URL (ending in .netlify.com) on the Netlify project overview.

Using your own domain name

Here again, Netlify eases the process, you simply need to have a domain name registered.

Once you have a domain name, you can carry on the settings:

Click on Set up a custom domain
In custom domain, click Add a domain alias and fill in the input with your domain name
Then you will need to point your domain to netlify servers, to do so, use the provided CNAME record in your DNS provider interface
After a few minutes to hours depending on your DNS provider, you should be able to access the website using your domain name.

Using TLS (HTTPS)

Once you can access the website using your domain, you will want to use https for security, SEO and performance purposes, you can and should use netlify https functionality:

Head to https in domain settings of your website:

Then click on Provision certificate
You will need to wait for the DNS update to enjoy your site in https

Deploy on content change

One last interesting feature is the ability to rebuild the website when content changes. As of now, if we add a new section on contentful, it's not reflected on the website. Let's change that so that any modification on the site triggers a rebuild of the website:

First head to the deployment settings in netlify, scrolling to build hooks
Clicking Add build hook, configure the name of your hook and the deployed branch:
Once saved, you need to copy the URL that you need to use to trigger the build:
Then go back to Contentful, going to webhooks in settings:
Click: Add a new hook
Name the hook as you like, then paste the netlify hook URL in the URL field:
Then you can either trigger the build for all event but I recommend limiting the hook to publish/unpublish actions to avoid useless builds.
Once the hook is saved, creating a new section will trigger a build on netlify:

Wrap-up words

Building a showcase website is mostly about content, presenting the products, services, the company goal/mindset. In this regard, using a CMS is the best way to enable your client to update their contents, keeping them happy

This article, was a simple introduction, to the currently available technologies enabling developers to build such a site. There are much more great features enabled by those tools, namely:

Working with images, videos, ...
Using different translations for contents
Workings with formatted texts without allowing contributors to inject html in the website

Thank you for getting this far and happy coding!

Don't lose sight of your bugs: How to improve your defect capture by 20%

Tue, 10 Mar 2020 00:00:00 GMT

Here we assume you correctly set up Sentry. Both your front-end and your back-end are able to send errors to Sentry and you set up Sentry releases (https://docs.sentry.io/workflow/releases/?platform=javascript).

Checkpoints

This is what we'll want to achieve

ALL unexpected behaviours (bugs) are captured in Sentry: no false negative
No expected behaviours are captured in Sentry: no false positive
Sentry issues contain all data needed to debug the issue

The first point is about identifying all bugs, the second about avoiding noise. This is all about making the workplace clean, uncluttered, safe, and well organized to help reduce waste and optimize productivity: see 5s.

Capture ALL exceptions

Why aren't all exceptions caught

Out of the box, Sentry works by listening to uncaught exceptions in your application. In addition, there are some integrations to listen to some technology-specific errors: eg redux-sentry-middleware.

Something to be aware of is that Sentry does not automatically capture caught exceptions: if you wrote a try/catch without re-throwing an exception, this exception will never appear in Sentry. Example:

async function getTodoList() {
  try {
    return await axios.get('/todos')
  } catch (error) {
    toaster.error('Error while retrieving todos')
    // `error` will never appear in Sentry! The only way for you to
    // know that this bug happens is waiting for users to report it,
    // and even then you'll have no data to debug it
    return []
  }
}

This is called "silencing" or "swallowing" exceptions.

Manually capture exceptions

The user experience of the code snippet above is fine: we show a toaster and do not crash the entire application. The only problem is with reporting: we need to know that this bug happened.

Sentry gives you an easy tool for this, that allows you to manually capture exceptions:

async function getTodoList() {
  try {
    return await axios.get('/todos')
  } catch (error) {
    Sentry.captureException(error) // this exception will now show up on Sentry
    toaster.error('Error while retrieving todos')
    return []
  }
}

Summary: exception handling rule

When you catch an exception, you have 2 exclusive choices:

re-throw it (throw error in Javascript)
Sentry.captureException

If this exception is a "normal" behaviour, make it clear with a comment. This should be very rare, you shouldn't use exceptions as "normal" control flow! But Python/Django sometimes do not leave you much choice, sadly.

Capture exceptions at the right place

This mostly applies to backend.

When an exception happens, it's usually quite deep down the call stack: for example accessing the database in a Django model's method called by a service called by a Django view. You want all exceptions to be captured, so you have 4 options here:

Let the exception propagate until it crashes your request (HTTP 500 internal error) or Celery task. That would be automatically caught by Sentry
Capture the exception as close to its source as possible: in the model's method
Capture the exception where there's most business logic context: in the service
Capture the exception as close to the app's edges as possible: in the Django view

Remember the exception handling rule: any exception caught should be either re-thrown, or manually captured. This means that you don't really have any choice:

An exception usually is due to an internal error: it should absolutely crash your request or your Celery task. It is important so that your logs and metrics are accurate, don't hide problems to yourself.
If for some reason you do not want the exception crashing your request/celery task (maybe it's something not-so-important that is acceptable to fail), you want to handle it as close to the edge of your application as possible: in the Django view for example. This allows all intermediate functions to have a chance to enrich the exception (raise SomeException from err in Python, or Sentry breadcrumbs) and is a good practice to avoid your core services making too much assumptions.

Remember that the exception handling rule is an either/or: you should not both capture the exception and re-raise it. The following is wrong:

def getById(id):
  try:
    User.objects.get(id=id)
  except Exception as e:
    Sentry.capture_exception(e) # This is wrong!
    raise UserNotFound from e # This is the exception that should be caught, higher in the call stack

It is wrong because it will lead to your exceptions being captured by Sentry multiple times, which means multiple Sentry issues for the same underlying problem, which means noise. Noise is bad, see 5S principles.

Capture non-exception bugs

Most bugs should trigger an exception (that you raise yourself if needed): invalid input, invalid state... But in some (hopefully rare) cases you might need to capture a bug in Sentry but still continue your work. For this, you can use Sentry.captureEvent() to capture fully-custom events.

This should be very rare: if you're in an invalid situation (bad input for example), probably you should raise an exception and let the caller handle it!

How to add extra data to Sentry events

Capturing the information is nice, but sometimes you have some data available in the code that you'd like to make available to Sentry for later debugging. Make extensive use of it: this makes the difference between an instant fix and a half-day debugging session because you can't reproduce a bug!

Breadcrumbs

Breadcrumbs are basically info logs. They are not errors themselves: they are only sent to Sentry when an error is actually captured. It allows you to get more info on what happened in the lifecycle of a request (backend) or in the user journey (frontend). You can probably assume that every log should also be a breadcrumb.

They are very easy to use, see the Sentry docs

function onLoginSubmit(event) {
  // ...
  Sentry.addBreadcrumb({
    category: 'auth',
    message: 'User logged in ' + user.email,
    level: Sentry.Severity.Info
  })
}

Context

You can set some "context" (set of key-value pairs, called "tags" or "extra data") to send to Sentry if and when an error is captured. For example, you might want to send the current user ID, the ID of the model targeted in a Django view, the referrer header, a tracing ID... For this you can use configureScope (https://docs.sentry.io/enriching-error-data/context/?platform=javascript#extra-context).

By default, the scope of the context is the entire request (backend) or lifetime of the app (frontend). You can set a scope local to a block of code (https://docs.sentry.io/enriching-error-data/scopes/?platform=javascript#local-scopes)

Extra data

Arbitrary key-value pair, that you can see on Sentry issues. You cannot filter issues based on these.

Example: value of local variable, message, timestamp...

Summary

Never swallow exceptions
Never captureException AND raise
In the backend, captureException as close to the edge as possible (Django views, Celery tasks)
Use configureScope extensively to add data to errors captured by Sentry

Master SQLAlchemy Relationships in a Performance Friendly Way

Mon, 09 Mar 2020 00:00:00 GMT

In a web application, performance is as important as any functional feature. In my current project, we experienced troubles with this aspect as we handled big databases with complex models and relationships. It worked fine when we tested the product with future users. But as soon as it got into production, the volume of data increased significantly. That's when the performance issues came out. We learned some useful stuff while fixing those issues. I will share them with you in this article.

First, each access to the database costs time, so you want to emit as few SQL statements as possible. The key to keeping this number stable when your data volume increases is to have a constant number of SQL statements emitted no matter the number of objects retrieved.

When handling objects with relationships, the query result is composed of nested objects and arrays. We learned the hard way that in those cases, serializing the query result to python object is an expensive operation. The easiest way to reduce the cost of serialization is to only retrieve the objects you need, nothing more.

Using the right loading technique for your relationships will help you reduce the number of SQL requests you emit. Also, it will allow you to retrieve only the relationships you need at the moment. Combined with better handling of query filters, it will allow you to reduce the serialization time of your queries.

1. Limit the number of queries emitted

SQLAlchemy provides several loading techniques to retrieve your relationships. You can specify the one you want to use when declaring a relationship, with the parameter lazy.

Loading techniques

relationship(ProjectModel, lazy="select")

There are 6 different loading methods. Here are, in my opinion, the 2 more useful :

Lazy loading: This technique is the default one. It will be used if you omit the lazy parameter in your relationship or if you specify lazy="select". This way of loading emits a separate query when first accessing the attribute.

class Parent(Base):
    __tablename__ = 'parent'

    id = Column(Integer, primary_key=True)
    child_id = Column(Integer, ForeignKey('child.id'))
    child = relationship("Child", lazy="select")

# emits a SELECT statement to get the parent object
parent = session.query(Parent).first()

# emits a second SELECT statement to get the child object
child = parent.child

Eager loading: The eager loading technique, also called joined loading, specify that you always want to retrieve the relationship. When accessing the parent element, SQLAlchemy will also join the relationship table. This way, when accessing the relationship attributes, it doesn't emit another SQL statement. The relationship is loaded when accessing the parent object.

class Parent(Base):
    __tablename__ = 'parent'

    id = Column(Integer, primary_key=True)
    child_id = Column(Integer, ForeignKey('child.id'))
    child = relationship("Child", lazy="joined")

# emits a SELECT statement to get the parent object and its children
parent = session.query(Parent).first()

# does not emit a second SELECT statement as child object is already loaded
child = parent.child

⚠️ If you want to use eager loading on a self-reference, you have to tell SQLAlchemy how many levels deep it should join. To do that, just add the parameter `join_depth` to the relationship declaration. Here is an example :

class Person(Base):
    __tablename__ = 'person'

    id = Column(Integer, primary_key=True)
    manager_id = Column(Integer, ForeignKey('person.id'))
    manager = relationship("Person", lazy="joined", join_depth=1)

person = session.query(Person).first()
person.manager # eager-loaded (one level deep)
person.manager.manager # lazy-loaded (two levels deep)

Choosing the right loading technique depends on when and how you use your relationships. As I said earlier, you want the number of requests to remain constant when the data volume evolves. That is why using the default loading technique can be bad for your app performance. If you retrieve multiple objects and their relationships, it is better to use the eager loading technique, as you will emit only one SELECT statement, no matter the number of objects you retrieve.

The trick here is that when developing a feature, you often manipulate a small number of objects. The lazy loading technique gets slow only when reaching a certain amount of data. So, measure the execution time with a realistic data volume. Also, if you know that the number of objects you manipulate will grow in time, you would better use the eager loading technique so the number of queries stays constant.

Be careful though not to use the eager loading technique too much. As I said before, retrieving useless objects is expensive. That's why I rarely choose this loading technique directly on the relationship declaration. I prefer to choose it on a query level, only when I'm sure I need the relationship. I will explain in the next section how you can achieve that.

2. Do not retrieve useless objects

As I said before, the serialization of query results in python objects is expensive. We noticed that executing the same SQL query on the database and with the ORM did not take the same time. It was faster on the database than with the ORM. The difference between the two is that the result on the database is raw data, while with the ORM you get nested objects. So we deduced that this was indeed the serialization that was increasing the request time.

There are two main things you can do to limit that :

Only retrieve object relationships when you need them. The deeper you go on the data you retrieve, the longer it will take to serialize the query result. You can do that by choosing your loading technique on a query level.
Do not wait to have python objects to filter the one you need. Likely, you can directly filter the data in the query, even if you need to filter by a relationship attribute.

To illustrate those two points, I will use the following example. You can see two models with one-to-one relationships, including a self-referential relationship.

class Person(Base):
    __tablename__ = 'person'
    id = Column(Integer, primary_key=True)
    firstname = Column(Text)
    lastname = Column(Text)

    address_id = Column(Integer, ForeignKey('address.id'))
    address = relationship('Address')

    manager_id = Column(Integer, ForeignKey(person.id))
    manager = relationship('Person')

class Address(Base):
    __tablename__ = 'address'
    id = Column(Integer, primary_key=True)
    number = Column(Integer)
    street = Column(Text)
    city = Column(Text)
    country = Column(Text)

Choose the loading technique on a query level

If you look at the example, you will notice that I did not specify a loading technique on the relationships. As I mentioned earlier, it is better to declare the loading technique in the query than in the relationship. This way, you can use different techniques depending on your needs. It will prevent you to retrieve relationships when you don't need them just because you set eager loading as the default technique.

Imagine you usually manipulate the Person object without their managers. The lazy loading is a good loading technique as you won't retrieve relationships you do not need. But then you want to return the list of all Persons with their managers for some purpose. You will want to eager load the managers with the persons so that you only make one SELECT statement. Here is how to do it :

Person.query.options(joinedload("manager"))

The joinedload option here has the same effect as if it was set in the relationship declaration. The difference is just that is will take effect only in one query, so you can use only if you need to get the relationship along with the main object.

The load options you can use in a query are the same that you can use in a relationship declaration. This allows you to have your default loading technique in the relationship but choose another in a query if needed.

Filter your queries by relationship attributes

Let's use the example from above. Imagine you want to get all the Person instances that live in Paris. You may get all the instances and then filter them with a python function. This is not the best way to do it. Filtering in the SQL query will be faster. First, because this language is optimized for filtering and other query operations. Also, the more objects you get, the longer it will take to serialize them to python objects.

Filtering by relationship attributes is quite simple. The only thing you have to do is to join your relationship before filtering, like this:

Person.query.join(Person.address).filter(Adress.city == 'Paris').all()

Now, you may encounter the need to filter on a self-referential relationship. To achieve that, you have to alias at least one side of your expression, so you can reference the table unambiguously. There are two ways of achieving that.

The first way is a bit verbose but clear and very flexible. You first define an alias for the table, here Person. By using it on the join, you will be able to explicitly tell which side of the relationship you are referring to when accessing attributes.

from sqlalchemy.orm import aliased

Manager = aliased(Person)
Person.query.filter(Person.firstname = 'Pauline').\
                join(Manager, Person.manager).\
                filter(Manager.firstname = 'Pierre').\
                all()

The second option is less verbose, but I find it confusing. In the join, you can use the property aliased. With that property, all use of the Person entity after the join will refer to the alias. You lose some flexibility as the sequencing of the functions is important here.

Person.query.filter(Person.firstname = 'Pauline').\
        join(Person.manager, aliased=True).\
        filter(Person.firstname = 'Pierre').\
        all()

Bonus

To help you improve your app performances, you may want to spot where you are lazy loading some relationships. If you use the raise loading strategy, with lazy="raise" in your relationship, SQLAlchemy will raise an error each time you try to retrieve a relationship without eager loading it first. This can be helpful to be sure you keep a constant number of SQL statements on big objects you manipulate.