Theodo

Why Aws Step Functions Is Not the Best Tool for Business Processes

Wed, 27 Jan 2021 00:00:00 GMT

I recently worked on a project where we needed to orchestrate a process, applying for loan to be precise, which could take weeks. To do this, we decided on using a serverless architecture and specifically AWS Step functions. The use of such a framework helped us immensely to organize the flow of the process but it wasn't without a few bumps along the road. My aim in this article is to warn you about those bumps and help you have a smoother ride.

I. AWS stacks and their limits

1. Cloud-formation stack

Cloudformation is a service from AWS which allows you to create and manage linked AWS resources (which are themselves other services). All the resources that we used in our project were initially in a single CloudFormation stack. Let's look at an example of a stack :

Here we can find our Lambdas, as well as Logs and if we scrolled further we could also find the state-machines we used. In terms of code, a stack corresponds to a configuration file called serverless.yml. In this file all the resources , environment variables, and outputs, are defined.

2. Cloud formation stacks limits

A stack has multiple limits which can quickly be reached. You can find the complete list in the AWS docs but we will only talk about the one encountered in the project.

A stack can contain a maximum of 200 resources. 200 state machines and Lambdas would indeed be a lot but there are other resources we have to take into account. Indeed, for each Lambda we create we also create a Log group resource and a Version resource. State Machines also have a Role resource. There are other resources we use like SQS, S3 buckets, DynamoDB tables. As mentioned above we had all our resources in a single stack. As you can already guess we reached that limit of 200 resources. We were unable to develop any new features until we found a solution to this limit. As such, we were hard-pressed to find a solution.

The first thing we thought of was, is this really a limit? Could our dev ops team set this limit to higher value or could we even contact the AWS support to ask for a less restrictive limit? We tried but to no avail. This limit is one you will have to live with and can't cheat your way around. Ok, so the quick fix we hoped for is a bust, let's get thinking about real solutions.

We knew we could split the stack into multiple ones but that would require a lot of refactoring and we already saw glimpses of the problems it would generate. So we searched for a smarter approach and we found a plugin that did exactly what we wanted: serverless-plugin-split-stacks, or so we thought. Three of us tried getting the plugin to work like we wanted without success. But what really made us give up on this solution was that we realised more and more that the real problem was the faulty architecture of our serverless application. Even if we could fix the resource problem with the plugin (if we got it to work) other problems might arise from the fact that we bunched up everything into one stack. So ultimately we decided that the pragmatical solution was to split our stack into multiple sub stacks.

3.Difficulties with concurrent stack updates

I'll spare you the details of how we split the stack, but we essentially had different services which we put in separate stacks and kept one main stack with nothing more than the orchestration between the different services. You can see in the image below an example of the split works. What I want to talk about here is the new problem brought on by the split.

For the split to work we have to export the resources we intend to use in the main stack, namely the different lambdas and sub state-machines. AWS allows us to do this quite easily by defining exports with region specific names that we can then import in another stack. The problem comes when we want to update an exported resource. Before going into more detail you will need to understand how AWS updates a resource. There are two types of updates :

Updating only the core of the resource, for example changing the code of a lambda
Updating the configuration of the resource, for example the name of the resource

You can find out more about Cloud Formation resource updates here. In the first case, AWS can update the resource without changing anything but the changes we asked for. In the second case AWS has to replace the resource with a new one. In order to do so, it has to delete the resource and create a new one.

Now we can come back to our problem. What happens when an exported resource needs to be removed to update it? Well AWS won't allow you to deploy your stack because you will be trying to remove a resource currently used by another stack. Quite the pickle. The only way to go around this is to duplicate the resource, apply the update on the new copy, change the import to use the new copy and finally remove the old copy. We realised this problem existed before we went ahead with splitting the stacks. We wagered that those updates would seldom be needed. I can confirm that a few months later those updates were very rare, and the proposed workaround worked, so all in all it was a good call. I choose to talk about this problem because it brings me to a larger underlying problem: handling versions in an AWS serverless framework.

II. Problems caused by differences in versions between state machines and lambdas

1.UseExactVersion

The problem we encountered with the limits of the stack brought us to another problem : handling changes in version. For me this problem is at the core of what I want to talk about in this article. We knew of it before but until this point it wasn’t too problematic. Indeed AWS gives us a tool to handle changes in version. For example : what happens when you deploy a new state machine with one lambda updated? Do the old state machines use the new lambda when they get to the step in question?

To answer these questions AWS gives us the very useful useExactVersion. By setting this parameter to true in the configuration of a step we tell AWS to synchronise the version of the state machine with the lambda called in the step. This way AWS keeps a version of lambda for each version of currently running state machines. To answer our initial question : no, AWS will not call the new lambda for old state machines. So, as I said earlier, with this tool we hadn’t really encountered big problems linked to versioning. But of course just like the stack had its limits so does useExactVersion.

2. Secondary state-machines

UseExactVersion works for lambdas called in a step. What happens when a step calls another state machine? In that case useExactVersion does not apply. Say we update state machine B which is called by state machine A. An old state machine A in version 1 will call a state machine B in the new version 2. If you have breaking changes between the versions, then you will have a problem.

For example, we wanted to sanitize the inputs of our whole process. To do so, we need to change the inputs of all our steps. For the steps calling lambdas there was no problem thanks to useExactVersion but the whole process still crashed because old sub state machines were expecting the complete input instead of the sanitized version. The only way we could proceed was to duplicate the whole state machine like we did when we wanted to split the stacks. This would lead to a third concurrent stack, which we judged too costly and lead us to abandon the idea of sanitizing the inputs.

3. Modifying the steps

That is not the only problem we encountered with versioning. Adding a new step to an existing state machine is not a problem and modifying the execution of a step is also feasible. But what happens when you want to delete a step? We thought about it and concluded that if we kept the code for the lambda that was called, it wouldn’t be a problem. Indeed, already running state machines would still have the definition of the step and since the code for the called lambda still exists, everything would be fine. Of course it wasn’t.

To be able to execute certain tasks, to call lambdas for example, AWS uses roles. For a step to be able to call a specific lambda function it needs the associated role. What we did not anticipate was that when we removed the step from the configuration of the state machine, the role generated for that step was also removed. That means that when an old state machine tried to call the removed step it did not have the role necessary. We had two options:

Replace the automatic generation of roles and manually add the needed roles in the configuration (over a hundred roles)
Keep the step in the state machine and put it somewhere it would not be reached. Although the second option felt like hiding the mess under the rug, we were pressed for time, so we decided it was the more pragmatic solution.

III. Conclusion

With these specific examples to illustrate the difficulties of versioning with AWS Cloudformation and Step Functions, I want to more generally show you that state machines are made to be deployed rarely. If you want to deploy a new version it is best to deploy a whole new state machine rather than update the current one. Now that is quite a limiting aspect, since we often work with products that are already in production, and we continuously upgrade them. If you need to use a state machine like architecture and can't afford to seldom deploy new versions I do have a recommendation. In a new project we have been using Camunda which allows you to code state machines on your Springboot (a Java framework) server. I found that it has many advantages over AWS. Mostly it has three functionalities I like :

It is less complex than AWS (no roles for example) and has a gui to create the state machine
It allows you to migrate state machines from one version to another if there are no breaking changes
It has an integrated REST engine that allows you to easily manipulate the state machine (change the current step, stop/start state machines etc.)

With that said, I hope this article either helped you on your journey with AWS and State machines or better prepared you to better start using these technologies.

Managing multiple loading states in Angular

Fri, 22 Jan 2021 00:00:00 GMT

When developing an Angular application, you will probably have to perform HTTP requests. Whether it be fetching data from a separate backend application or an external api, your application might have to wait for a response at some point.

Properly handling the state of an HTTP request in a web application is a common use case. In a large codebase, it quickly becomes a critical matter of architecture. This matter is even more important than Angular is a verbose framework.

In this context, your solution to this concern has to be :

Simple enough to keep your codebase readable
Reusable enough to keep your codebase sane and to save time when creating new pages or components
Adaptive enough to handle the different use cases of your application

This article aims at presenting several ways of dealing with a request's loading or error in Angular, with regards to these three aspects.

Why should we care about showing error messages and loaders

The first main reason for having a hard time on this, is because we care about our users. We want them to understand easily what is going on the application, and we don't want them to click ten times on a button while waiting for an async operation to be performed.

Thus, it is important to give them as many feedback as we can when we perform async stuff.

RxJS

Angular asynchronous execution management essentially lean on reactive programming, using RxJS. Thus, every pattern presented here involve this library.

In this article, I will mention Observables and BehaviorSubjects , if you are not familiar with reactive programming and especially with these 2 objects, I invite you to check out the very well designed documentation of the library!

Managing one request state using the component state

Principle

The first and easiest way to keep tracks of your requests is to directly watch it within your component.

Let's say we build an api for displaying cute little pictures of cat breeds.

Our component declares two boolean attributes for storing the current state of the request (loading and error) and these attributes are updated throughout the data fetching operation:

What it looks like in a component

@Component({
  selector: 'app-fetch-breeds',
  templateUrl: './fetch-breeds.component.html',
})
export class FetchBreedsComponent {
  constructor(private myService: MyService) {}

  loading = false;
  error = false;
  errorMessage = '';

  fetchBreeds() {
    // Reset request error state data
    this.error = false;
    this.errorMessage = '';

    // Set the component in a loading state
    this.loading = true;

    // Start the fetch event pipeline involving :
    this.myService.fetchBreeds().pipe(
      // A handler which is called when our service triggers an error and
      // which is dedicated to setting the error in a corresponding state
      catchError(err => {
        this.error = true;
        this.errorMessage = err.message; // Or whatever error message you like
      })
      // A callback which is always called whether an error has been triggered
      // or not.
      // It is responsible for setting the component in a non-loading state.
      finalize(() => {
        this.loading = false;
      })
    )
    .subscribe();
  }
}

<div>
  <!-- If we are fetching the data, we display a loader -->
  <app-loader *ngIf="loading"></app-loader>
  <button (click)="fetchBreeds()">See these magnificent cats!</button>
  <!-- If there is an error (and we are done waiting for a response), we display it -->
  <app-error-message *ngIf="!loading && error">
    {{ errorMessage }}
  </app-error-message>
</div>

Managing request state within a service

Why delegating this logic to a service ?

Using services in an Angular application offers many advantages (DRY principle, efficient testing among others). In our case, it also allows us to persist and share data between components / other services.

It also allows abstraction: your component is dedicated to the UI display while a service handles any inner complex logic of the component.

Case 1 - The state is shared between multiple components

Let's say we have two components offering two features based on the cats entity: on one page we see the pictures of the breeds and on the other page we can see some technical details about these breeds.

This architecture involves the duplication of the logic responsible for sending the request, catching errors, and handle a loading state in between.

This logic could be moved into a dedicated service, which will expose one observable for the loading information and one for the errors:

@Injectable()
export class CatBreedsService {
    constructor(private http: HttpClient) {}

    private _loadingSubject = new BehaviorSubject<boolean>(false)
    isLoading$: this._loadingSubject.asObservable();

    // We expose observables because we want our components to have a read access
    // but no write access to the information

    private _errorSubject = new BehaviorSubject<string | null>(null);
    erros$: this._errorSubject.asObservable();

    fetchBreeds(): Observable<CatBreed[]> {
        this._loadingSubject.next(true);
        this._errorSubject.next(null);
        return this.http.get('/cat-breed').pipe(
            catchError(err => {
                this._errorSubject.next(err);
                return of([]);
            }),
            finalize(() => this._loadingSubject.next(false))
        )
    }
}

Now, the components only have to call the fetch helper from the service, and the loading / error state information about the entity will be shared across the entire application.

@Component({
  selector: "app-list-cat-breeds",
  templateUrl: "./list-cat-breeds.component.html",
})
export class ListCatBreedsComponent implements OnInit {
  constructor(private catBreedsService: CatBreedsService) {}

  isLoading$ = this.catBreedsService.loading$;
  hasErrors$ = this.catBreedsService.errors$;

  ngOnInit() {
    this.catBreedsService.fetchBreeds.subscribe();
  }
}

Managing multiple request states within a service

Case 2 - Concurrency Issues

Example use case: multiple occurrences of an autocomplete input

You might also encounter the case where you have to perform a request for the same operation but in multiple contexts :

For instance, let's imagine we have a form where each input dynamically fetch autosuggestion data on change, and show a loader while the update is performed. If we create unique subjects for handling the state of this request, there will be concurrency issues.

A concrete exemple of this in our context would be a page where we can attribute breeds to cat pictures.

The problem with side effects

In this case, since there is only one loading subject for every input, a change on any input will put all the other ones in their loading state or error state.

The same happens with the suggestions observable. It triggers unnecessary renderings in your application, which may have catastrophic consequences for the performance of the application.

Side effects mean you have to break down this component into multiple parts

One could handle this issue by having a complex state including a loading observable and an error observable. The code beneath shows such an implementation :

export interface InputState {
  loading$: BehaviorSubject<boolean>;
  error$: BehaviorSubject<string >;
}

@Component({
  selector: 'app-cat-breed-form',
  templateUrl: './cat-breed-form.component.html',
})
export class CatBreedFormComponent implements OnInit {
  inputs = ['input1', 'input2', 'input3'];
  inputsState: { [key: string]: InputState } = {};

  ngOnInit() {
    this.inputs.forEach(input => {
      this.inputsState[input] = {
        loading$: new BehaviorSubject(false),
        error$: new BehaviorSubject<string | null>(null)
      };
    }, {});
  }

  setLoading(inputId: string, value: boolean) {
    this.inputsState[inputId].loading$.next(value)
  }

  setError(inputId: string, value: string | null) {
    this.inputsState[inputId].error$.next(value)
  }

  getLoading(inputId: string) {
    return this.inputsState[inputId].loading$.asObservable()
  }

  getError(inputId: string) {
    return this.inputsState[inputId].error$.asObservable()
  }

  updateField(inputId: string) {
    this.setLoading(inputId, true)
    this.project$.pipe(
      take(1),
      catchError(err => {
        this.setError(inputId, err)
      }),
      finalize(() => this.setLoading(inputId, err))
    ).subscribe(() => { // do your stuff })}
  }
}

A few remarks :

The component logic is getting intricate.
You can't deport the fetch logic into a service, or at least you have to manage the loading/error state from within the component.
In order to have a readable template, one will probably have to implement getters/setters

thus increasing the code written in the component ts file.

Breaking this down into simpler components, which in this case means implementing a field component, we are back to the first case with no concurrency.

We can create a dedicated service which exposes a registration function which :

takes an id in input
creates a formControl on which the 'change' event triggers a search
returns the formControl and the associated loading, error, and autocomplete suggestions observables.

@Injectable()
export class SearchService implements OnDestroy {
  constructor(private http: HttpClient) {}

  subscriptions: Subscription[] = [];

  registerControl<T>(id: string, initialvalue: T) {
    const control = new FormControl(initialvalue);
    const _loadingSubject = new BehaviorSubject<boolean>(false);
    const _errorsSubject = new BehaviorSubject<string | null>(null);
    const _suggestedSubject = new BehaviorSubject<T[]>([]);
    this.subscriptions.push(
      control.valueChanges
        .pipe(
          switchMap((query: string | null) => {
            if (query !== null && query.length > 0) {
              return this.searchOnQuery<T[]>(query).pipe(
                catchError(err => {
                  _errorsSubject.next(err);
                  return of([]);
                }),
                finalize(() => _loadingSubject.next(false))
              );
            }
            return of([]);
          }),
          tap(suggestions => _suggestedSubject.next(suggestions))
        )
        .subscribe()
    );

    return [
	control,
	_loadingSubject.asObservable(),
	_errorsSubject.asObservable(),
	_suggestedSubject.asObservable()
    ];
  }

  private searchOnQuery<T>(query: string) {
    return this.http.get<T>('/search', {
      params: {
        query
      }
    });
  }

  ngOnDestroy() {
    this.subscriptions.forEach(sub => sub.unsubscribe());
  }
}

Now we just have to register any new input within this service :

@Component({
  selector: 'app-cat-breed-autocomplete-input',
  templateUrl: './cat-breed-autocomplete.component.html',
  styleUrls: ['./cat-breed-autocomplete.component.scss']
})
export class CatBreedAutocompleteInput implements OnInit {
  @Input() initialValue: Entity;
  @Input() autocompleteId: string;

  control: FormControl = new FormControl();
  loading$: Observable<boolean> = of(false);
  error$: Observable<boolean> = of(null);
  suggestions$: Observable<boolean> = of([]);

  constructor(private searchService: SearchService) {}

  ngOnInit() {
    const [control, loading$, error$, suggestions$] = this.searchService.register<CatBreed>(
      this.autocompleteId,
      this.initialValue
    )
    this.control = control;
    this.loading$ = loading$;
    this.error$: error$;
    this.suggestions$ = suggestions$;
  }
}

These 4 observables are now available in the template and can be used with async pipes.

This allows every input to behave independently, since they have their own event management system through the FormControl API

Conclusion

In this article we have seen two different ways of handling the loading state of a request using RxJS, and how to incorporate this within your Angular architecture.

With only one loader to handle in one component, it is okay to implement the UI logic in the component. With multiples loaders at the same time, things become easily messier and we saw how to handle this new complexity in the case of multiple autocomplete inputs.

How to create a video calling app using webRTC and websockets

Fri, 22 Jan 2021 00:00:00 GMT

With lockdowns and restrictions, the past year has not been great for social interactions. This might be the perfect time to build your first video call app, or to add a video chatting feature to your existing project.

If you are unsure on how that works or are having some troubles setting it up, you came to the right place. In this article I will show you how to create a very simple video calling application using react (yes, with Typescript) and SpringBoot. If you are not using this tech stack, fear not, the principles are the same and will apply to whichever framework you want to use.

At the basis of a video call app there are two things we need to use and learn: peer-to-peer network (P2P) and websockets. Let's dive right into them!

What is a peer-to-peer network (P2P)?

You are probably familiar with client-server networks. In this model, computers are connected to a server and can only communicate to it (send or receive data). If two computers (A and B) wants to share data, they have to pass through the server:

This seems a bit inefficient, wouldn't it be easier to cut off the middleman and just let A and B talk to each other? Yes, it would, and this is what is called a peer-to-peer communication. The information is directly shared by the computers:

This can be achieved very easily using WebRTC (Web Real-Time Communications). WebRTC is an open-source framework that allows real time communication between browsers. It was released in 2011 and is supported by Google Chrome, Safari, Mozilla Firefox, Microsoft Edge, Opera, Brave and Vivaldi.

There are also javascript libraries that simplify the creation and usage of webRTC connections. The most popular ones are easyRTC, PeerJs, RTCPeerConnection, SimpleWebRTC and SimplePeer. As shown below, simple-peer is by far the most used one, and it is the library that we will use for the rest of this tutorial.

A webRTC connection takes place with a process called signaling:

Peer A (the initiator) decides to create a connection, an offer signal is automatically created
The offer is given to peer B
Peer B connects to it to create an answer signal
The answer is given to peer A
Peer A connects to it to complete the peer-to-peer connection

What exactly are these offers and answers? These are simply objects which contain two parameters: type and sdp. The type defines whether the object is an "offer" or a "answer". The sdp (Session Description Protocol) contains the information about the connection, such as the IP Address, the port number, and the media to be shared.

If you haven't already asked yourself, there is a slight problem with the process described above. How will user B receive the offer? And how will user A receive the answer? This is where websockets come into play.

What is a WebSocket?

WebSocket is a communication protocol, just like the famous http. The main difference is that while an http protocol is unidirectional, a webSocket is bidirectional. Let's see what this means and how they differ.

With http, a user can make a request to a web server (http://example.com). When this is done a connection is created, the server gives back a response, and once the response is received the connection is closed. This protocol is unidirectional because the sender has to create a request to receive a response. If new information is available, a new request has to be made to obtain it.

With webSockets (ws), a user can also make a request to a web server (ws://example.com). The difference is that when this is done a handshake is made between the user and the server which creates a connection that will never be closed, unlike the http. This ever-lasting connection can be used for bidirectional communication (without requiring the user to always have to make a new request to open a new connection). If new information is available after the original request, it will be sent to the user through the connection.

This is a very important difference, crucial when we want to have access to real-time data. When checking the score of sport match online, nobody wants to have to refresh the page every 5 second (to make an http request and open a new connection) to check whether the score has changed. With a webSocket every time the score changes the server will automatically update it on your browser.

This is exactly how we will share the webRTC offer and answer signals. Whenever user A (the initiator) creates an offer, he will send it through the websocket connection and it will automatically be received by user B (who will then do the same with the answer payload).

The final architecture

The diagram below shows what the final architecture of the app will be.

The two peer signal each other by sending and receiving the offer and answer signals. This will be done using the WebSocket. After this signaling process, the peer-to-peer connection is established and the two peers can share the webcams media streams to video call each other. Now let's get started.

Setting up the WebSocket

The first step is to create the webSocket server, in our spring boot application project. SpringBoot has a sping-WebSocket module which makes the whole set-up extremely simple as we will see.

In the pom.xml file, add the spring-boot-starter-websocket dependency as follows:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-websocket</artifactId>
</dependency>

Now we can create a class annotated with the @Configuration and @EnableWebSocket:

@Configuration
@EnableWebSocket
public class WebSocketConfiguration implements WebSocketConfigurer {

    @Override
    public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
    registry.addHandler(new SocketHandler(), "/videochat").setAllowedOrigins("\*");
    }
}

The class implements WebSocketConfigurer and overrides the registerWebSocketHandlers method, which is used to add websockets endpoints. In this case we add a websocket endpoint "videochat" and with a handler given by the SocketHandler class. The latter, which we will create next, defines how all the sessions and messages coming into ws://localhost8080/videochat should be handled.

The final step is to create the class SocketHandler with annotation @Component:

@Component
public class SocketHandler extends TextWebSocketHandler {

  List<WebSocketSession> sessions = new CopyOnWriteArrayList<>();

  @Override
  public void afterConnectionEstablished(WebSocketSession session) {
    sessions.add(session);
  }

  @Override
  public void afterConnectionClosed(WebSocketSession session, CloseStatus status) {
    sessions.remove(session);
  }

  @Override
  public void handleTextMessage(WebSocketSession session, TextMessage message)
  throws IOException {
    for (WebSocketSession webSocketSession : sessions) {
      if (!session.equals(webSocketSession)) {
        webSocketSession.sendMessage(message);
      }
    }
  }
}

Let's see what is going on above. The class extends TextWebSocketHandler for which we are overwriting three methods:

afterConnectionEstablished is called every time a browser completes a handshake with the webSocket and creates a connection/session (remember this connection will last forever until either parties closes it). The session is appended to the sessions list.
afterConnectionClosed works similarly: when a browser closes a connection this method is called and the session is removed from the sessions list.
handleTextMessage is called whenever a particular session sends a message to the webSocket. When this happens we will iterate through all the sessions connected to the webSocket and send the message to each of them except for the one whose message came from (to avoid sending a message to yourself).

Making the peer connection

Now that the websocket server is done, we can move on to the frontend to build the actual video calling application. We will create a component which return the following content:

enum ConnectionStatus {
 OFFERING,
 RECEIVING,
 CONNECTED,
}

export const VideoCall = () => {
  const videoSelf = useRef<HTMLVideoElement | null>(null);
  const videoCaller = useRef<HTMLVideoElement | null>(null);
  const [connectionStatus, setConnectionStatus] = useState<ConnectionStatus | null>(null);
  const [offerSignal, setOfferSignal] = useState<SignalData>();

  return (
    <div className="web-rtc-page">
      {connectionStatus === null && (
        <button onClick={() => sendOrAcceptInvitation(true)}>CALL</button>
      )}
      {connectionStatus === ConnectionStatus.OFFERING && (
        <div className="loader"></div>
      )}
      {connectionStatus === ConnectionStatus.RECEIVING && (
        <button onClick={() => sendOrAcceptInvitation(false, offerSignal)}>
          ANSWER CALL
        </button>
     )}
     <div className="video-container">
       <video ref={videoSelf} className="video-block" />
       <video ref={videoCaller} className="video-block" />
     </div>
   </div>
 );
};

The page contains two video elements for the user and for the receiver, these have a useRef hook each, which will be used later on to dynamically add the webcam streams. The page also contains a state for the connection status, based on which some elements will be conditionally rendered:

null: when the page is first rendered. A button is present which can be clicked to start the call. This calls the function sendOrAcceptInvitation on click which we will define later
offering: when a user has started a call and is waiting for the answer on the other side. A loader is displayed while waiting.
receiving: when the user receives a call. A button is present to answer the call, this also calls sendOrAcceptInvitation on click, but with different arguments
connected: when the peer-to-peer connection is completed

Now it's time to create the function sendOrAcceptInvitation:

const [simplePeer, setSimplePeer] = useState<Instance>();

const sendOrAcceptInvitation = (isInitiator: boolean, offer?: SignalData) => {
  navigator.mediaDevices
    .getUserMedia({ video: true, audio: false })
    .then((mediaStream) => {
      const video = videoSelf.current;
      video!.srcObject = mediaStream;
      video!.play();

      const sp = new SimplePeer({
        trickle: false,
        initiator: isInitiator,
        stream: mediaStream,
      });

      if (isInitiator) setConnectionStatus(ConnectionStatus.OFFERING);
      else offer && sp.signal(offer);

      sp.on("signal", (data) => webSocketConnection.send(JSON.stringify(data)));
      sp.on("connect", () => setConnectionStatus(ConnectionStatus.CONNECTED));
      sp.on("stream", (stream) => {
        const video = videoCaller.current;
        video!.srcObject = stream;
        video!.play();
      });
      setSimplePeer(sp);
    });
};

In this function we create the simple-peer instances. The arguments of the function are isInitiator (true for the user making the call and false for the user receiving the call) and offer (the offer with the sdp of the caller, only present for the user answering).

The first step is to access the webcam media stream and set it to the videoSelf reference. Next a simplePeer instance is created, with the webcam video as the stream and with initiator value set to true/false depending if this is the caller or the recipient.

The simple-peer instance has certain events which can be fired, for which we will define the respective methods. In this case we have:

signal: when the simple-peer has a signal ready to be sent (the offer or answer). These are the signals that we have to give to the other users though the webSocket (step 2 and 4 of the signaling process). Remember that for the initiator this signal (the offer) is created automatically. A non-initiator, however, needs to connect to the offer signal in order to generate the answer (step 3). This is exactly what the line sp.signal(offer) is for
connect: when the connection between two peers is complete
stream: when a stream is received from the other peer. In this case, each peer will receive the webcam stream of the other user which was passed during the initialisation of the simplePeer instance. Once this is received, we set it to the videoCaller reference

We are almost there, the last step is to create the connection to the webSocket and define what should be done when the signals are received.

const webSocketConnection = new WebSocket("ws://localhost:8080/videochat");

useEffect(() => {
  webSocketConnection.onmessage = (message: any) => {
    const payload = JSON.parse(message.data);
    if (payload?.type === "offer") {
      setOfferSignal(payload);
      setConnectionStatus(ConnectionStatus.RECEIVING);
    } else if (payload?.type === "answer") simplePeer?.signal(payload);
  };
}, [simplePeer]);

Creating the connection to the websocket is very straightforward, we simply create an instance of the WebSocket class, with the endpoint url as its argument.

We can use the onmessage property to catch all of the incoming offer/answer signals (which were sent using webSocketConnection.send() in the sendOrAcceptInvitation function). When receiving the offer we set the signal as a state and change the peerConnectionState, this will render the acceptance button and the user will be able to connect to the signal and only if desired. When receiving the answer signal, however, we automatically connect the initiating simple-peer to the other: successfully closing the connection.

Congratulations! The peer-to-peer connection is now established and the users are able to talk to each other despite the lockdown.

Wrapping it all up

As you saw it is relatively easy to create a video calling application. We were able to create a simple one in just about 100 lines of code! With simplePeer.send and simplePeer.on("data", ()=>{}) we can even send/receive messages and upgrade our application to a video calling AND messaging app.

I hope the tutorial was useful, you can find the full code here.

Why Rome Tools Isn't Ready to Replace Eslint, Webpack and Babel... Yet

Fri, 18 Dec 2020 00:00:00 GMT

Introduction

While building your React, Angular, or Vue project, you may have wondered why you needed to set up and configure Babel, Webpack, ESLint, and Prettier separately and tried to look for an existing bundled implementation of all the features these tools provide. That is when you might have stumbled upon Rome Tools. I will show you in this article some of the best features Rome provides compared to ESLint as well as how I tried migrating my ESLint project to Rome:

TLDR

Rome has great features including explicit diagnostics, provides a VSCode extension which makes it a promising tool. However, only the linter and formatter are available and stability is a major pain point. In addition to this limitation, the lack of configuration options can scare off developers.

The Rome philosophy

Rome's ambition is to unify tools that have previously worked as separate. It wants to have strong conventions with minimal configuration. It aspires to go further than create-react-app or vue-cli as these tools just include multiple other tools with their dedicated configuration files. Rome aims at providing a single frontend development toolchain for all the following features:

Bundling
Compiling
Documentation Generation
Formatting
Linting
Minification
Testing
Type Checking

So far, only the linter is supported by Rome Tools and it was built to provide features that ESLint has been lacking for some time, that is displaying diagnostics. I've used ESLint in many projects and I've always been annoyed by how poor the error messages displayed are. Rome tries to tackle that and I find that the diagnostics displayed with rome check are really helpful to understand the error, how you can fix it, and what you can do to avoid it appearing in the future:

Rome does a great job at displaying diagnostics!

Rome also includes a watch mode, and review to allow you to apply safe fixes, comment to disable the rule, ignore the issue, etc... all of that in its comprehensive CLI:

Diagnostics, advice, and fix options

Rome is also shipped with its file history. Say you apply an autofix on all your files but forgot to exclude a directory, you can easily recover your formatted files with rome recover pop. I find this command truly handy, especially when setting up your formatting or auto-fixing your issues.

Experimenting and migrating from ESLint

To build up a better judgment of Rome Tools, I took one of my existing projects that were using ESLint and tried migrating it to Rome which made me encounter most of the problems Rome has.

The first thing I did was to add Rome to my project. Applying formatting is a cakewalk with Rome: running rome check --apply and Rome will format and apply safe fixes to your entire codebase. I then started working on fixing the linting errors and that's when Rome hit me hard. Most of my React components were written as:

const Component = (props: Props) => {};

Rome has a rule against this (useFunctionDeclarations). It requires that you use function declarations instead of constant declarations:

function Component(props: Props) {}

I'm not going to go into details but I find that first syntax handy. In most of my projects, it allows me, at a glance, to quickly distinguish utils functions from components that return JSX. So I decided I wanted to remove that rule and that's when I discovered that configuring your linting rules is impossible to do with Rome. The contributors chose to go with a zero-configuration stance. The only thing that can be configured is the paths you want to ignore in the configuration file:

lint: {
  ignore: ["src/__mocks__/"];
}

You can also suppress a rule on a specific line with an explanation but it's far from ideal as these comments will undermine actual explanations comments in your code. The lack of configuration was my first pain point while experimenting with Rome.

I still tried to fix all my linting errors while working with Rome with the CLI. While it seems promising, there are yet too many errors that cannot be fixed. Hence, the review CLI tool can prove a little useless when the only option it's offering is to comment out the error.

I also had the case where the tool would present me with the same error over and over again until I realized the suppression comment was just not working properly:

return <Style.Animate className={animation}>
		// rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading // rome-ignore lint/jsx/noPropSpreading
		<BaseComponent {...props} />

Rome offers an extension to use in one IDE: Visual Studio Code (there is no plan to bring the equivalent to Webstorm for the moment). I was really happy when I discovered that because I rely a lot on my IDE to apply formatting, quick fixes, read lint errors output, etc...

Overall, I can say that the extension does also an excellent job at displaying diagnostics. The information that is available in the CLI is also available when hovering on an error:

Explicit diagnostics straight in VSCode

Most of the safe fixes, as well as auto-formatting, can also be applied straight from your IDE without having to use the CLI, which I greatly appreciated. There are a few exceptions to this rule, for instance, I was not able to format my tsconfig.json using the extension but could do it from the CLI.

However, I've been struggling a lot with that extension as well as with the CLI while migrating from ESLint to Rome as they both appear to lack stability. I had more than 200 files to migrate to Rome and I ran into a huge amount of crashes while doing so which made my experience with Rome pretty painful. I find it discouraging to have to restart my IDE or the CLI review because I keep getting exceptions:

! Rome exited as this error could not be handled
and resulted in a fatal error. Please report if necessary.

✖ Connection died

Looking at the number of issues related to the extension or the CLI, we can tell that stability is a major area of improvement for Rome.

Conclusion

I find it amazing that projects such as Rome emerge to tackle the weaknesses of predominant tools such as ESLint and offer alternatives. Rome tries to do that and I can only encourage these kinds of initiatives.

The fact that only the linter/formatter is available isn't a major issue in my opinion. Even if it means that Rome isn't a bundled implementation yet, it is a choice they made to have a part of Rome ready as quickly as possible. Nevertheless, it lacks both configurability and stability. I would consider that point a red flag due to how it slowed me down while migrating. Rome wants to replace all the other tools but, paradoxically, makes it harder by not providing configurability.

Finding out why your code isn't working and how you can improve your development practices is something Rome definitely provides but even with that, I cannot recommend starting a project with Rome at this day. I'm hoping that the Rome developers will stabilize it, offer configurability and implement the rest of the features Rome wants to provide to offer a developer-friendly monolith.

Get started with Django and Jupyter Notebooks on VSCode in minutes

Thu, 17 Dec 2020 00:00:00 GMT

This is a guide primarily for developers familiar with Django, looking to add a data analytics or data science element to their web development project.

Don't have much time and comfortable with Jupyter? Click here to see the initializer file you should create + import into every python script you want to run: Final Code

Want to see an example data analytics script you can use for any Django project? See here: Example Script

Data Science for Web Apps

Django is a (very popular) Python-based framework that helps you build web apps frighteningly quickly. A quick Google/YouTube search for Django tutorials and you'll have an avalanche of content coming your way.

Jupyter Notebook is a tool that lets you build python scripts fast. It's especially popular for data science prototyping as it lets you break up your code and re-run parts of your script.

Why merge the two?

Today, data analytics is (and should be) the mantra behind every successful product owner's launch plan - and as a developers, providing guidance and infrastructure to enable this is extremely valuable. However, combining these two in one stack can lead to some interesting errors and so, I've written this article to help you cross these hurdles quicker than I did.

Prerequisites

At this point, you should have a Django development server setup in a virtual environment (if not, there's plenty of tutorials for that). This setup uses MacOS/OX (though OS shouldn't matter) with VSCode.

If you don't have Python already set up in VSCode with a good formatter and would like to, click here.

Configuring Django to run Python scripts without the server

Start by creating a folder inside your Django root directory named python_functions. All your python scripts can live in here.

At this point, if you try and import something from Django (and you don't have your development server running), you'll get an error that your settings are not configured. To see the error, type the following into hello.py and run your script:

  from django.contrib.auth.models import User

You should see an error message that ends with:

  django.core.exceptions.ImproperlyConfigured: Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

This happens because without running manage.py, Django doesn't configure and run your project - and so doesn't know where to import modules from.

To fix this, we'll need to firstly specify an environment variable to our settings file and then call the setup function. As we'll have to do this for every python script we write, let's do all the setup in a separate script, and import it into hello.py.

Creating an initialiser script

Create a new file in the python_functions directory called django_initializer.py. In this file, we first set an environment variable pointing to our settings.py file, then run the function to setup Django.

But we're not done yet! As django_initializer.py isn't in the project's root directory, python won't recognise "arcwebsite.settings" as a legitimate path. To fix this, we need to manually add the project root directory to the python path.

Altogether, here's what we've got in django_initializer.py:

  # A script that's needed to setup django if it's not already running on a server.
  # Without this, you won't be able to import django modules
  import sys, os, django

  # Find the project base directory
  BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

  # Add the project base directory to the sys.path
  # This means the script will look in the base directory for any module imports
  # Therefore you'll be able to import analysis.models etc
  sys.path.insert(0, BASE_DIR)

  # The DJANGO_SETTINGS_MODULE has to be set to allow us to access django imports
  os.environ.setdefault("DJANGO_SETTINGS_MODULE", "arcwebsite.settings")

  # This is for setting up django
  django.setup()

Remember, this is separate to the actual file we want to run, as seen below:

Make sure the initialiser script is in the same folder as your actual script

Run hello.py once more, and you should see it work with no problems!

Now, you've got a setup to run Python scripts to analyse your data, that can interact with your Django database even when your server isn't running. This is especially useful when prototyping on a local machine before deploying any scripts to production.

The final step is to set up Jupyter Notebook in VSCode, to interact with your Django project.

Setting up Jupyter Notebook in VSCode for a Django Project

Why Jupyter Notebook?

Jupyter Notebook is a tool by Project Jupyter to make data science modelling and prototyping easy. One of the main benefits is the ability to split large procedural programming scripts into individual lines or blocks, written in 'cells', and run them one at a time.

An example use of the cell-by-cell feature of a Jupyter Notebook

Running cells individually has a lot of benefits - for one, you can split your script into data collection, pre-processing, model training, and prediction steps, and run these segments individually. This can be especially useful when tweaking hyper-parameters on your models for example (or for any other instances you may want to re-run just one section of code from your script).

Jupyter Notebooks can of course be run in the browser, rather than in VSCode, using the following command:

  (myenv) $ python manage.py shell_plus --notebook

With this however, you wouldn't get the benefit of other VSCode extensions on your ipynb file, and easy access to your non-python files within the same code editor. Thus, setting up Jupyter inside VSCode makes our dev experience that little bit better.

Installing Jupyter Notebook in Django

So let's get started. First we'll need to install Jupyter and django-extensions. django-extensions is a collection of custom extensions for the Django Framework. It's often needed to support many other custom packages such as this one.

Run this command in your terminal:

  (myenv) $ pip install jupyter django-extensions

Next, add django_extensions to your INSTALLED_APPS in settings.py and restart the django server:

As we're already using VSCode with the Python extension, we can create a new Notebook quite easily. Simply bring up the command bar (cmd+shift+P) and select in 'Create New Blank Jupyter Notebook'.

After opening your first .ipynb file (the file extension for Jupyter Notebooks) you should get a VSCode warning prompting you to install an ipykernel. Click on 'install' when this pops up. Once installed, you won't need to do this again for any subsequent notebook files you create in the same virtual environment.

NB: This will change how your shell looks when you run python manage.py shell on the command line - but unless you've manually customised the default Django shell to your liking, this shouldn't be a problem.

Now, copy over the contents of hello.py (the two imports) into the new .ipynb file, and press run - your code should run without any errors!

Now let's try a simple queryset operation. Add a line to display the first user on your system (or an equivalent queryset operation):

  print(User.objects.all()[0])

There's an error!

Running queryset operations with the default setup causes errors

On closer inspection, you should see the error message ends with:

  SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.

Why does this error occur?

Skip to the next section if you'd just like the fix!

Django was originally built with the WSGI standard in mind - a standard for synchronous Python web servers, frameworks and applications. With the release of Django 3.0 came support for ASGI, a standard for asynchronous applications, allowing developers to create applications designed to run in an async context.

However, certain key parts of Django rely on a global state being maintained, which is not coroutine-aware (i.e. not able to run in an async context). The most prominent example of this is Django's ORM (object-relational mapper) - essentially, its queryset filtering features, which allows you to interact with an SQL database through Python.

Running any of these sync-only Django functions in an async context gives you the SynchronousOnlyOperation error. The ideal way to deal with this is to wrap your function in Django's sync_ to _async(), but for a Jupyter Notebook where the async context is forced onto us by the environment (not our code), we need to manually allow sync-only operations, by setting DJANGO_ALLOW_ASYNC_UNSAFE to true. The Django documentation also has a section explaining async safety and has stated that async support for the ORM will be available in future releases.

Fixing the SynchronousOnlyOperation Error

The quickest way to fix this is by setting another environment variable. Go to django_initializer.py and add:

  os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"

Which means django_initialiser.py in its final state now looks like:

# A script that's needed to setup django if it's not already running on a server.
# Without this, you won't be able to import django modules
import sys, os, django

# Find the project base directory
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

# Add the project base directory to the sys.path
# This means the script will look in the base directory for any module imports
# Therefore you'll be able to import analysis.models etc
sys.path.insert(0, BASE_DIR)

# The DJANGO_SETTINGS_MODULE has to be set to allow us to access django imports
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "arcwebsite.settings")

#  Allow queryset filtering asynchronously when running in a Jupyter notebook
os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"

# This is for setting up django
django.setup()

Finally, reload the window and run the ipynb script again. You should get a success!

Finishing Touches

To recap - we've configured Django to give us access to the framework without the server running and set up a dev environment with Jupyter Notebooks for easy prototyping and testing of data analytics scripts.

With these tools, you should now be able to run Python scripts inside of VSCode with familiar data science packages such as scikit-learn, pandas, and matplotlib, just as you would with a regular Jupyter Notebook - as well as have access to your Django database with queryset filtering.

Good luck!

Start Tracking Website Metrics - An Example

Here's an example script to track the total number of weekly user sign ups on your Django app from a given start date. Note that there are some dependencies required that haven't been mentioned in this tutorial, but after installing them, you should be good to go!

# Imports
import django_initializer

from django.contrib.auth.models import User
from analysis.models import WorkoutEntry
from datetime import datetime, timedelta
import pytz
import time
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)

# Set a start date
start_date = datetime(2020, 7, 13, tzinfo=pytz.UTC)
today = pytz.utc.localize(datetime.today())

# Setup for weekly metrics
week_start = start_date
weeks = []
while week_start <= today:
    weeks.append(week_start)
    week_start += timedelta(days=7)
    week_strings = [week.strftime("%d/%m") for week in weeks]

# Count number of cumulative sign ups
tot_users = [] # total users
pc_growth = [] # percentage growth week on week
for (index, week) in enumerate(weeks):
    weekly_signups = User.objects.filter(is_active=True).filter(date_joined__gte=start_date).filter(date_joined__lt=weeks[index])
    tot_users.append(len(weekly_signups))
    if index > 1:
        pc_growth.append(np.around((100* (tot_users[index] - tot_users[index-1]))/(tot_users[index-1]), 2))
    else:
        pc_growth.append(0)
users_per_week_df = pd.DataFrame({"tot_users":tot_users, "pc_growth":pc_growth}, index=week_strings)

print(np.mean(np.around(pc_growth, 2)))
print(users_per_week_df)

fig, axs = plt.subplots(figsize=(10,6))
axs.plot(users_per_week_df["tot_users"])
axs.set_xticks(axs.get_xticks()[::4])
axs.set_xlabel("Date")
axs.set_ylabel("Total Users")
axs.set_title("Total User Signups",{'fontsize': 18, 'fontweight': 600})
axs.grid()
plt.show()

Setting up Python in VSCode with the Black Formatter [optional]

We'll be adding the Python extension to VSCode, setting a virtual environment and setting up a formatter that you can customise to your preference.

The first step is to ensure a great developer experience for programming in Python in VSCode. For this, you'll first need the VSCode Python extension, which you can find by simply searching for 'Python' in the extension sidebar. As of June 2020, there's also the new 'Pylance' extension, which is also suitable.

Download this and reload the developer window (cmd+shift+P, type in 'reload' and you should see the command) and open a .py file. You should then see the Python version and interpreter in the bottom left corner of the screen. If it doesn't show your chosen virtual environment settings, click on it to update the settings to the interpreter of your choice (select from the dropdown or type in the path required).

Next, we'll set up the formatter - we're going to use Black (Github). As Black is usually installed and run via the command line, we'll have to do some extra work to integrate it into VSCode, and run every time we save a file.

There's no Black VSCode extension, so we'll have to install it the old fashioned way. In your terminal, run:

(myenv) $ pip install black

Next, open your workspace settings (cmd+shift+P, type "settings", and select 'Workspace Settings' without JSON). In the search bar at the top, search 'format on save', then check the checkbox:

Finally, type in 'python formatting provider' and select 'black' from the dropdown:

Now reload your window, create a .py file and each time you save, you should see the words 'formatting with black' flicker on and off on the bottom information bar (in line with your python interpreter details). The next step is optional, and for those who want to customise Black.

When running Black from the command line, you can pass in a list of arguments for custom preferences. To set these, go back into your VSCode settings and search 'black args'. You can then pass in individual strings. For example, to set maximum line length to 100 characters:

That's all you need - now your VSCode should be set up to code in Python, with automatic formatting on save.

Build your own stickers component with react-konva

Mon, 16 Nov 2020 00:00:00 GMT

Konva.js is an awesome HTML5 Canvas JavaScript framework that enables high performance animations for desktop and mobile applications.

The framework's capabilities are useful and versatile - it can be integrated for many different and varied use cases. Konva.js deals well with touch genstures, drag and drop and animations to name just a few!

To see the breadth of the konva.js framework see the konva.js demos page.

React-Konva

React-Konva allows us to work with both React applications and Konva.js together seamlessly.

Here we'll create draggable stickers which can be deleted, suitable for both desktop and mobile applications. There are great react-native stickers libraries out there, however there really aren't many libraries to implement it in a react app. Therefore, react-konva is a great solution!

Show me the code!

Check out my code sandbox for the demo here!

Let's get started...

Adding a konva image

Let's take this tutorial through step-by-step. We need a canvas to add our stickers to. We'll add an image here to replicate a selfie, the sticker's most common use case.

import React from "react";
import { Image as KonvaImage, Layer, Stage } from "react-konva";
import useImage from "use-image";

export default function App() {
  const [image] = useImage("example-image.png");

  return (
    <Stage width={500} height={500}>
      <Layer>
        <KonvaImage image={image} />
      </Layer>
    </Stage>
  );
}

What is going on here?

The useImage hook handles loading the image
The Stage acts as our canvas
The Layer is tied to the canvas element and can contain Groups, Shapes and Images
The KonvaImage acts like an img tag and allows us to display our image

Photo by Moose Photos from Pexels

Aside: When I was building this for a photobooth experience I utilised the react-html5-camera-photo library to take the photo that we used as the image in the canvas, but let's stick to the stickers for now 👀

Creating the sticker button

Let's add a button with our desired sticker. We'll click this button whenever we want to add a new sticker to the canvas.

<button
    className="button"
    onMouseDown={() => {
      addStickerToPanel({
        src: "sticker-1.png",
        width: 100,
        x: 100,
        y: 100
      });
    }}
>
  <img alt="Eiffel Tower" src="sticker-1.png" width="100px" />
</button>

We use the button's onMouseDown property to add the sticker to our array of images to be added to our canvas. We set the width and the x and y position where the sticker should appear on the canvas.

We initialise an empty array called images, using react's useState hook, and add the width and src properties to the array when we click a sticker!

const [images, setImages] = useState([]);

  const addStickerToPanel = ({ src, width, x, y }) => {
    setImages((currentImages) => [
      ...currentImages,
      {
        width,
        src,
        x,
        y
      }
    ]);
  };

Adding a draggable sticker

Let's create a separate component that contains our individual stickers which will be on the canvas.

We can only use konva components within our Stage in App.js so we use a Konva Image component.

import useImage from "use-image";
import React from "react";
import { Image as KonvaImage } from "react-konva";

export const IndividualSticker = ({ image }) => {

  const [stickerImage] = useImage(image.src);
  const stickerHeight = stickerImage
    ? (image.width * stickerImage.height) / stickerImage.width
    : 0;

  return (
    <KonvaImage
	  draggable
      width={image.width}
      height={stickerHeight}
      image={stickerImage}
      x={image.x}
      y={image.x}
    />
  );
};

Again, we load our image using the useImage hook
To get the image to scale properly we add the stickerHeight calculation
In the KonvaImage we add the x and y properties to determine where the sticker appears on the canvas when the button is clicked.
The draggable property allows us to drag the image within the canvas!

In App.js let's map over the images within our layer.

{images.map((image, i) => {
  return <IndividualSticker key={i} image={image} />;
})}

Note: The sticker will always appear on top of the image if the stickers are below the image in our file (so there's no need for a z-index).

Multiple Stickers

Let's add multiple different styles of stickers 🤩

Create a stickers data file to include an array of your chosen stickers. Include the url, width and alt name. Add as many as you like!

export const stickersData = [
  {
    url: "sticker-1.png",
    width: 100,
    alt: "Eiffel Tower"
  },
  {
    url: "sticker-2.png",
    width: 150,
    alt: "Statue of Liberty"
  },
  {
    url: "sticker-3.png",
    width: 60,
    alt: "Big Ben"
  }
];

In App.js import the stickers data and map over them to add all the buttons!

{stickersData.map((sticker) => {
        return (
          <button
            className="button"
            onMouseDown={() => {
              addStickerToPanel({
                src: sticker.url,
                width: sticker.width,
								x: 100,
								y: 100
              });
            }}
          >
            <img alt={sticker.alt} src={sticker.url} width={sticker.width} />
          </button>
        );
      })}

Checkpoint: Draggable stickers that work well on mobile and desktop! 🎉

⁉️ WAIT, but I want to be able to delete the stickers!?

✅ Okay, let's continue with the tutorial. Things get a little more complicated here!

Deleting the stickers

In order to delete stickers from the canvas we add a cross icon to the top right corner of each sticker.

On mobile the cross appears on long tap of the sticker
On desktop the cross appears when you hover over the sticker

Let's add another KonvaImage to our individual sticker and wrap it in a Group so the sticker and cross icon are grouped.

export const IndividualSticker = ({ image, onDelete, onDragEnd }) => {
  const [stickerImage] = useImage(image.src);
  const [deleteImage] = useImage("cancel.svg");

  const stickerWidth = image.width;
  const stickerHeight = stickerImage
    ? (image.width * stickerImage.height) / stickerImage.width
    : 0;
  return (
    <Group
      draggable
      x={image.x}
      y={image.y}
      onDragEnd={(event) => {
        onDragEnd(event);
      }}
    >
      <KonvaImage
        width={image.width}
        height={stickerHeight}
        image={stickerImage}
      />
      <KonvaImage
        onTouchStart={onDelete}
        onClick={onDelete}
        image={deleteImage}
        width={25}
        height={25}
        offsetX={-stickerWidth / 2 - 20}
      />
    </Group>
  );
};

Render a cross icon using the useImage hook
Add the draggable, x and y properties to the Group so that they are common to both images
Use the offset properties on the delete button to position the cross where you like
Add the onClick property and pass the onDelete function. Also add the onTouchStart property so that this works on mobile too!
Pass an onDelete function detailed in App.js
- Use a splice to remove the image from the local state when the delete button is tapped/clicked

onDelete={() => {
  const newImages = [...images];
  images.splice(i, 1);
  setImages(newImages);
}}

We need to ensure that the x and y position of each sticker in the local state is updated appropriately - we make use of the Group's onDragEnd property. We pass in the following function from the App.js file. This updates the x and y position of the sticker at the end of the drag movement.

onDragEnd={(event) => {
    image.x = event.target.x();
    image.y = event.target.y();
  }}

Checkpoint: The cross shows when we add a sticker to the photo. When we tap/click the icon, it deletes the sticker as expected

Now, we only want to show the cross when we hover on desktop or long press on mobile - let's continue!

We use the react-use useLongPress and useHoverDirty to show the cross on long tap (mobile) or on hover (desktop).

Let's amend IndividualSticker.tsx to look like the following...

const imageRef = useRef(null);
const isHovered = useHoverDirty(imageRef);

const [showDeleteButton, setShowDeleteButton] = useState(false);
const [isDragging, setIsDragging] = useState(false);

const onLongPress = () => {
    setShowDeleteButton(true);
  };
const longPressEvent = useLongPress(onLongPress, { delay: 200 });

useEffect(() => {
    if (isHovered) {
      setShowDeleteButton(true);
    } else {
      setTimeout(() => {
        setShowDeleteButton(false);
      }, 2000);
    }
  }, [isHovered]);

return (
    <Group
      draggable
      x={image.x}
      y={image.y}
      onDragStart={() => setIsDragging(true)}
      onDragEnd={(event) => {
        setIsDragging(false);
        onDragEnd(event);
      }}
    >
      <KonvaImage
        ref={imageRef}
        width={image.width}
        height={stickerHeight}
        image={stickerImage}
        {...longPressEvent}
      />
      {showDeleteButton && !isDragging && (
        <KonvaImage
          onTouchStart={onDelete}
          onClick={onDelete}
          image={deleteImage}
          width={25}
          height={25}
          offsetX={-stickerWidth / 2 - 20}
        />
      )}
    </Group>
  );

We add a ref to the image that will be hovered. useHoverDirty is true when the image is hovered
We store the isDragging and showDeleteButton booleans in the local state using the useState hook
When we long tap and image we set the showDeleteButton state to true after a delay of 200ms
The useEffect ensures that the delete button disappears after the 2s of the image being hovered if it is not deleted.
We only display the delete button if the the showDelete button state is true and isDragging is false. This ensures that we don't show the delete button when we're dragging. This is a little hacky but it considers all the cases for mobile and desktop use!

We now need to ensure that the cross disappears if a user taps an area not associated with the sticker.

In App.js

const addStickerToPanel = ({ src, width, x, y }) => {
    setImages((currentImages) => [
      ...currentImages,
      {
        width,
        x,
        y,
        src,
        resetButtonRef: createRef()
      }
    ]);
  };

const resetAllButtons = useCallback(() => {
    images.forEach((image) => {
      if (image.resetButtonRef.current) {
        image.resetButtonRef.current();
      }
    });
  }, [images]);

  const handleCanvasClick = useCallback(
    (event) => {
      if (event.target.attrs.id === "backgroundImage") {
        resetAllButtons();
      }
    },
    [resetAllButtons]
  );

We create a new ref on each button using React's createRef
We add the handleCanvasClick which resets all the refs on the buttons so that we don't display the cross!

In the Stage add the onClick and onTap to handle this.

<Stage
        width={600}
        height={400}
        onClick={handleCanvasClick}
        onTap={handleCanvasClick}
      >

The Result

Using react-konva to build a great photobooth experience within a react app was really fun and I'll look forward to using this framework again! Check out more react-konva examples here and don't forget to checkout my codesandbox!

Sources + links:

Create resizeable split panels in React

Mon, 09 Nov 2020 00:00:00 GMT

Resizable containers empower different users to customize a UI to emphasise what they find most important. A basic implementation of this is a split view or split pane, allowing users to enlarge content that is more relevant to them.

There are libraries that can implement this feature for you, but creating it yourself gives you full control over the styling and functionality, so you can rapidly tailor it to your project. The full feature can be done in under 150 lines, with no dependencies, and in my case it was faster to add to my project than using a third-party library. It's also a component that covers a wide range of the features React has to offer, which is great for quickly getting to grips with what the library can do.

You can find the finished component right here.

(Credit to Seif Ghezala for the initial inspiration I've expanded on in this article)

Getting started

Creating the component

Let's start out by creating a functional component. We want to be able to pass in the content to be displayed in each panel, so let's go ahead and pass in the LeftContent and RightContent as props, and wrap them in styled div so they're displayed side by side.

import React from "react";

interface SplitViewProps {
  left: React.ReactElement;
  right: React.ReactElement;
}

export const SplitView: React.FunctionComponent<SplitViewProps> = ({
  left,
  right,
}) => {
  return (
    <div className="splitView">
      {left}
      {right}
    </div>
  );
};

.splitView {
  height: 100%;
  display: flex;
  flex-direction: row;
  align-items: flex-start;
}

Now let's add the divider. I'm going to keep things simple and just use a div with a margin. The width will be fixed at a few pixels, and I want the height to grow with the content in the panels either side, which we can do with height 100%.

  return (
    <div className="splitView">
      {left}
      <div className="divider"/>
      {right}
    </div>
  );

.divider {
  width: 2px;
  height: 100%;
  margin: 1rem;
  border: 2px solid #808080;
}

Alright here's our basic component set up.

Now for the fun part!

Time to start resizing

So the end goal is to click and drag the divider and resize both panels accordingly. The trick here is to resize just one of the panels, and then let flex-grow and flex-shrink take care of the other.

Let's control the panel on the left. We're going to need a way of storing and setting its width so we can create some state for our component using the useState hook.

const [leftWidth, setLeftWidth] = useState<undefined | number>(undefined);

The panel on the left is going to have quite a bit of its own functionality so lets move that div into its own new component called LeftPanel. We want it to use the state we just created so lets pass in leftWidth and the setter too.

const LeftPanel: React.FunctionComponent<{
  leftWidth: number | undefined;
  setLeftWidth: (value: number) => void;
}> = ({ children, leftWidth, setLeftWidth }) => {

  return <div>{children}</div>;
};

Time for the magic. The ref!

React refs allow us to access the properties of elements in the DOM. If we use it on the div around our left panel content, we'll be able to both get and set the width of it directly.

To use refs, first create one using React's createRef function. Then attach it to the desired element using the ref property.

  const leftRef = createRef<HTMLDivElement>();
  return <div ref={leftRef}>{children}</div>;

By doing this, the properties of that element are now accessible under leftRef.current.

Time to connect our state and the ref with another React hook, useEffect. First we want to make sure that if our state isn't defined it gets initialized to the width of the left panel.

Then we want to make sure that whenever our state changes, we change the width of the left panel too. We can access the width using current.clientWidth.

const LeftPanel: React.FunctionComponent<{
  leftWidth: number | undefined;
  setLeftWidth: (value: number) => void;
}> = ({ children, leftWidth, setLeftWidth }) => {
  const leftRef = createRef<HTMLDivElement>();

  React.useEffect(() => {
    if (leftRef.current) {
      if (!leftWidth) {
        setLeftWidth(leftRef.current?.clientWidth);
        return;
      }

      leftRef.current.style.width = `${leftWidth}px`;
    }
  }, [leftRef, leftWidth, setLeftWidth]);

  return <div ref={leftRef}>{children}</div>;
};

Now whenever we change the leftWidth state in our original SplitView component, it will change the width of the left panel!

Notice the array after the contents of the useEffect. That dependency array makes sure that the code in the effect runs whenever either of those values change.

So we could do the same thing for the right panel, but as I said before, we can make things much simpler by giving it flex-grow (here I've used the shorthand flex: 1).

export const SplitView: React.FunctionComponent<SplitViewProps> = ({ left, right}) => {
  const [leftWidth, setLeftWidth] = useState<undefined | number>(undefined);
  return (
    <div className="splitView">
      <LeftPanel leftWidth={leftWidth} setLeftWidth={setLeftWidth}>
        {left}
      </LeftPanel>
      <div className="divider"/>
      <div className="rightPane">{right}</div>
    </div>
  );
};

.rightPane {
  flex: 1;
}

Now wasn't that easier?

We're all set to resize both of our panels. We just need something to trigger it.

Getting the mouse involved

We want to resize our panels when we click and drag the separator. We can break that down into 3 stages:

Clicking down on the separator
Dragging the mouse
Releasing the separator

Conveniently, we can capture each of these stages with 3 mouse events;

onMouseDown: Set a flag to start tracking mouse movements. Store the initial position of the mouse at this moment.
onMouseMove: If the flag is set, track the mouse's horizontal position, and resize the left panel accordingly
onMouseUp: Clear the flag

Let's start with onMouseDown. When the user clicks on the separator, we want to store the mouse's horizontal position at that moment, and set a flag to start tracking any horizontal movement after that. Sounds like we need some more useState hooks.

  const [separatorXPosition, setSeparatorXPosition] = useState<undefined | number>(undefined);
  const [dragging, setDragging] = useState(false);

Then we can create our onMouseDown function. We want to trigger this when we click on the separator, so let's add our onMouseDown event to the divider.

  const onMouseDown = (e: React.MouseEvent) => {
    setSeparatorXPosition(e.clientX);
    setDragging(true);
  };


  return (
    <div className="splitView">
      <LeftPanel leftWidth={leftWidth} setLeftWidth={setLeftWidth}>
        {left}
      </LeftPanel>
      <div className="divider" onMouseDown={onMouseDown}/>
      <div className="rightPane">{right}</div>
    </div>
  );

Next up is the onMouseMove event. If the flag isn't set we can just return. If the flag is set, then we're going to want to update leftWidth. The value to modify leftWidth by is the mouse's current position minus the stored mouse position. All we need to do is add that value to the leftWidth using the setter. We update the stored mouse position as well, so we can repeat this event for as long as the user is dragging the slider.

  const onMouseMove = (e: React.MouseEvent) => {
    if (dragging && leftWidth && separatorXPosition) {
      const newLeftWidth = leftWidth + e.clientX - separatorXPosition;
      setSeparatorXPosition(e.clientX);
      setLeftWidth(newLeftWidth);
    }
  };

To trigger this function, we need event listeners. We can add those to the document using document.addEventListener. If we create a new useEffect for our SplitView component, then the listener will be added when the component mounts. As an added bonus we can return a function in the useEffect, which is run when the component unmounts, to remove the eventListeners and clean up after ourselves.

  useEffect(() => {
    document.addEventListener('mousemove', onMouseMove);
    return () => {
     document.removeEventListener('mousemove', onMouseMove);
    };
  });

Alright almost there. Finally we just need an onMouseUp function to reset the flag. We'll add eventListeners to the document for onMouseUp events too.

  const onMouseUp = () => {
    setDragging(false);
  };

  useEffect(() => {
    document.addEventListener('mousemove', onMouseMove);
    document.addEventListener('mouseup', onMouseUp);

    return () => {
     document.removeEventListener('mousemove', onMouseMove);
     document.removeEventListener('mouseup', onMouseUp);
    };
  });

And we're done! Congratulations... but at the moment this component just gets a passing grade from me. Let's see what we can do to make it nicer.

Improving the component

Adding maximum and minimum widths

Because otherwise the component is really easy to break by losing the divider out of bounds and not being able to click on it.

To stop the divider from doing this, let's add a minimum width for our panels with a const (if you wanted to get fancy you could pass it in as a prop). Then we just need a bit of extra logic in our onMouseMove function.

const MIN_WIDTH = 50;

  const onMouseMove = (e: React.MouseEvent) => {
    if (dragging && leftWidth && separatorXPosition) {
      const newLeftWidth = leftWidth + e.clientX - separatorXPosition;
      setSeparatorXPosition(e.clientX);

      if (newLeftWidth < MIN_WIDTH) {
        setLeftWidth(MIN_WIDTH);
        return;
      }

      setLeftWidth(newLeftWidth);
    }
  };

So that takes care of the left edge, but what about the right? Well if we had the width of the whole component then we could calculate a maximum width for the left panel and use the logic above. Now how could we access the width of our component…

That's right, using another ref! Let's attach it to the div wrapping all our components.

const splitPaneRef = createRef<HTMLDivElement>();
...
  return (
    <div ref={splitPaneRef} className="splitView">
      <LeftPanel leftWidth={leftWidth} setLeftWidth={setLeftWidth}>
        {left}
      </LeftPanel>
      <div className="divider" onMouseDown={onMouseDown}/>
      <div className="rightPane">{right}</div>
    </div>
  );

Now our onMouseMove logic becomes:

  const onMouseMove = (clientX: number) => {
    if (dragging && leftWidth && separatorXPosition) {
      const newLeftWidth = leftWidth + clientX - separatorXPosition;
      setSeparatorXPosition(clientX);

      if (newLeftWidth < MIN_WIDTH) {
        setLeftWidth(MIN_WIDTH);
        return;
      }

      if (splitPaneRef.current) {
        const splitPaneWidth = splitPaneRef.current.clientWidth;

        if (newLeftWidth > splitPaneWidth - MIN_WIDTH) {
          setLeftWidth(splitPaneWidth - MIN_WIDTH);
          return;
        }
      }

      setLeftWidth(newLeftWidth);
    }
  };

Perfect, no more disappearing dividers.

What is this? A divider for ants?

It's not ideal that our user has to click on an area 2 pixels wide if they want to resize the panels. Maybe we could make it wider?

That looks terrible. How about padding it a little with another div?

  return (
    <div className="container">
      <LeftPanel leftWidth={leftWidth} setLeftWidth={setLeftWidth}>
        {left}
      </LeftPanel>
      <div className="divider-hitbox" onMouseDown={onMouseDown}>
        <div className="divider"/>
      </div>
      <div className="rightPane">{right}</div>
    </div>
  );

.divider-hitbox {
  cursor: col-resize;
  align-self: stretch;
  display: flex;
  align-items: center;
  padding: 0 1rem;
}

Muuuuch better. Changing the cursor also makes it much clearer to the user that this is resizeable!

This is really important for touch screens where 2 pixel precision isn't going to make you many friends.

But none of my touch screen users can use the feature!

That's right! Because we only added mouse events. We're going to need some touch events too!

TouchEvents are pretty similar to mouseEvents so we can reuse most of our code. But since touch screens can support many touches at once, touchEvents store an array of touches. Each touch has its own x and y positions, so we need to tweak our functions a little to account for this.

  const onTouchStart = (e: React.TouchEvent) => {
    setSeparatorXPosition(e.touches[0].clientX);
    setDragging(true);
  };

  const onMove = (clientX: number) => {
    if (dragging && leftWidth && separatorXPosition) {
      const newLeftWidth = leftWidth + clientX - separatorXPosition;
      setSeparatorXPosition(clientX);

      if (newLeftWidth < MIN_WIDTH) {
        setLeftWidth(MIN_WIDTH);
        return;
      }

      if (splitPaneRef.current) {
        const splitPaneWidth = splitPaneRef.current.clientWidth;

        if (newLeftWidth > splitPaneWidth - MIN_WIDTH) {
          setLeftWidth(splitPaneWidth - MIN_WIDTH);
          return;
        }
      }

      setLeftWidth(newLeftWidth);
    }
  };

  const onMouseMove = (e: MouseEvent) => {
    if (dragging) e.preventDefault();
    onMove(e.clientX);
  };

  const onTouchMove = (e: TouchEvent) => {
    onMove(e.touches[0].clientX);
  };

  const onMouseUp = () => {
    setSeparatorXPosition(undefined);
    setDragging(false);
  };

  useEffect(() => {
    document.addEventListener('mousemove', onMouseMove);
    document.addEventListener('touchmove', onTouchMove);
    document.addEventListener('mouseup', onMouseUp);

    return () => {
      document.removeEventListener('mousemove', onMouseMove);
      document.removeEventListener('touchmove', onTouchMove);
      document.removeEventListener('mouseup', onMouseUp);
    };
  });

  return (
    <Container ref={splitPaneRef} className={className}>
      <LeftPanel leftWidth={leftWidth} setLeftWidth={setLeftWidth}>
        {left}
      </LeftPanel>
     <div
        className="divider-hitbox"
        onMouseDown={onMouseDown}
        onTouchStart={onTouchStart}
        onTouchEnd={onMouseUp}
      >
        <div className="divider" />
      </div>
      <div className="rightPane">{right}</div>
  );
};

We can move out the core logic of the onMouseMove function into one shared onMove function that both the onTouchMove and onMouseMove functions can use. Also note that we use the onTouchStart and onTouchEnd events, both attached to the Separator, and reuse the onMouseUp function for onTouchEnd.

One last note is the addition of e.preventDefault if dragging is true in onMouseMove. This stops text being highlighted (or other onDrag mouse events from being triggered) while moving the divider around.

That's great! Can I style it too?

You can! The final touch to our component will be adding a className prop. Drop the component anywhere in your project and style it to your heart's content using your method of choice, whether it be stylesheets, styled-components or something else.

interface SplitViewProps {
  left: React.ReactElement;
  right: React.ReactElement;
  className?: string;
}

export const SplitView: React.FunctionComponent<SplitViewProps> = ({ left, right, className }) => {
...
  return (
    <div ref={splitPaneRef} className={`splitView ${className ?? ""}`}>
>

Happy resizing!

(Cover photo by Fabrizio Verrecchia on Unsplash)

Manage your technical debt roadmap right from your code 🚀

Wed, 21 Oct 2020 00:00:00 GMT

Exec summary:

Technical debt leads to bugs creation: the number of bugs and the number of design flaws (technical debt) are 0.92 correlated in a study from the Software Engineering Institute
Static analysis tools like SonarQube are massively used (100k+ users) to chase quality defects, but they can't replace human intelligence
Bug tracking system are time-consuming and don't help developers while they code
Our tool Tyrion fixes these problems and allows developers to see quality problems right in their IDE

Quality is worth its investment

Martin Fowler in his famous article on quality explains why investing in software quality is cheaper than not investing in it.

Why quality in software is cheaper than no quality

This knowledge has also long been known for lean practitioners: "If you focus on quality, you will deliver a high quality product on time. If you focus on delivery, you will deliver a low quality product late."

Why is that? If you focus on delivering things right the first time you will have less rework to do and then save time in the long run. Teams that don't do that end up spending most of their time fixing bugs.

Low quality leads to too many bugs and missed deadlines

Static analysis tools are mandatory but not enough to monitor quality

Once you know you want to deliver quality on the long run, there are a lot of things to do: identifying best practices, tech trainings, problem solving, quality tooling, setting up all kind of monitoring, choosing quality KPIs ...

If you are serious about keeping the quality of the codebase high, you need to proactively chase, monitor and resolve code quality defects. Even if you have set up all the best linters rules and have a highly trained team, in the long run you will have defects. Indeed, there are a lot of defects that linters can't detect: non homogeneous code and architecture, wrong patterns usage, naming not coherent with business...

A widespread strategy is to monitor quality by static analysis tools like SonarQube or CodeClimate. These tools have some advantages:

✔️You get a quality score without extra work from a developer (once the tool has been set up which can be quick)
✔️Results are objective
✔️They give you a list of piece of code you need to change and which good practices have not been followed

You can have graphs like this one which give you the trend of your technical debt and show you if the situation is getting better or worse.

A SonarQube debt graph from a current project

When talking with tech leads and CTOs, many of them told me that SonarQube has been efficient to help their team keep the quality of their code high.

Static Code Analysis Tools (SCAT) provide objective metrics and insights of the code quality and technical debt. However, these tools require a real integration effort. Such tools without a team adoption and training are of little value. On the opposite, used by trained developers, with a pragmatic understanding of the metrics and warnings, these tools offer a lot of clues on the next refactoring needed to reduce project entropy, on the security breaches still to address and on the potential bugs to review. SCAT warnings and suggestions must be used as starting points of developers' discussions around quality standards. SCAT must not be considered as referees stating good/bad quality points of a project or a team.
— Guillaume Michel, Theodo, VP Engineer

It has also been proven to be effective in some studies like Oswal, Nikhil. "Technical Debt: Identify, Measure and Monitor." (2019)

However, those tools have also limitations:

❌ they create quite a lot of false positive (labeling a piece of code as bad when it is good) and false negative (missing some bad piece of code) requiring times to sort everything.
❌ they can't inspect all aspects of a project. For example some defects like a wrong or non homogeneous directories' organisation are never detected

Academic research while confirming the positive impact of static analysis tools insist that they are not mature enough to only rely on them. For example, to assess if a defect will create a bug, Valentina Lenarduzz warns us in her 2019 study "Are SonarQube Rules Inducing Bugs?": "SonarQube violation severity is not related to the fault-proneness and there-fore, developers should carefully consider the severity as decision factor for refactoring a violation."

This is exactly what Bill Clark explained very well in his article "A TAXONOMY OF TECH DEBT", human can provide much more valuable information about a piece of code with technical debt like the contagious aspect: "Is it likely this bad code practice could spread somewhere else by copy/paste?"

Performant quality monitoring requires human analysis

As we just see, human analysis provides complementary information to static analysis tools:

✔️human can analyse every aspect of the project: architecture, naming, easiness of understanding, maturity of library and frameworks
✔️they can provide much more information about the problems they detect since they understand the context around the codebase
✔️they can provide a prioritization of the problems

The more the developers in your teams are trained about writing quality the more efficient they will be to detect defects. This is another reason why you should invest a lot in training if you want to achieve high quality in your code.

You also need to agree on the best way to write things: what is OK and what is not OK to write in the code. For example, these pieces of code are doing the exact same thing in Symfony.

/**
* @ORM\Column(unique=true)
*/
private $email;


/**
* @ORM\Column(type="string", length=255, unique=true)
*/
private $email;

Both ways have their advantages: the former is shorter to write/read and the latter is more explicit. I prefer the explicit version because I think in this case the tradeoff between the code being super clear and having more words to write and read is worth it. Some other people could say that it is not worth it to add the type and length because it's the default.

Who is right is less important than having your team agreeing in the standard way of doing it. Because once you agreed, people are autonomous to write and check quality on their own regarding this standard making little by little the whole code base coherent.

The main drawback of human analysis is that they require a lot of effort to get and update the information about the overall quality.

Comparison between static analysis and human analysis for software quality

Existing tools are time-consuming and don't help devs when they are coding

At Theodo we experienced several systems to track self admitted technical debt (SATD, as software quality researchers calls it).

On small or short projects, we use a special column in Trello/Jira/Monday/Google doc to track tasks we need to do to fix quality problems. This works OK as long as the team fixes everything quickly, so the column doesn't get too long and messy.
When the project grows we quickly need a real tech roadmap on a separate board.

Example of debt management using Trello

We can also use a specialized bug tracking tool like Bugzilla or Bugherd. It works well, but it has two drawbacks:

❌ it is time-consuming to update it when an item has been fixed or is no longer valid
❌ developers don't see the defect in their IDE while coding and thus could copy/paste it creating the contagious aspect of the debt explained by Bill Clark.

Introducing Tyrion: a tool to help teams track software quality through human analysis

I initiated Tyrion to remove those main drawbacks and be able to:

✔️Document the debt while coding directly in the IDE
✔️Painlessly update the debt tracking system when something has been fixed
✔️Quickly get an overview and the trend of the debt

Writing comments in the code to map the debt

Instead of registering all technical defects in a tool, the main idea of Tyrion is to add them directly in the code base as comments. Tyrion will parse these comments to generate the debt graphs or CSV files so you can analysis the debt.

The syntax of the comment is straightforward: the only thing you need to add to a classic comment is a type to allow Tyrion to group them:

// TODO DEBT_TYPE "Author: comment"

I like the following example from a current project. Albéric and Emyly are working on improving the accessibility of the webapp. Nothing urgent from the business and nothing requires immediate change from a technical point of view. But still, they know they can do this piece of code better, so they mark it as debt to improve it a bit later.

// TODO accessibility "Alberic: Add keyboard behaviours and focusability"
export const Select: React.FC<Props> = ({

Even if you don't use Tyrion, adding debt comments in the code is valuable by it-self. It trains developers to evaluate the quality of the code while they are coding. It also partially protects you again the contagious effect of the debt because if you have correctly marked with comments the debt in the code, developers will think twice before copy/pasting a piece of code that is marked as a debt.

If you have a large code base, it can take a while to document the whole technical debt. You can do it gradually during few weeks. While working on some part of the app each member of your team will read different files and find something to mark as debt. By doing that you will create little by little the debt map of your project. Of course, it's better to fix a problem when you see it but for various reason you don't always have the time to do it.

If your project is of reasonable size, you can also organize a big debt mapping session one afternoon.

Using the CLI to generate the first graph

Once you have commented at least a part of the debt you have in your project, you can start using the Tyrion CLI.

First you need to download it with: npm i -g tyrionl or yarn global add tyrionl

Then just run tyrion in the root directory of your project. It will scan your files to look for TODO, FIXME or @debt strings in the comments and then assume it is a debt comment.

This first mode is useful to get information of the current debt status of your project. It can help you respond to question such as "How much debt do we have right now?" or "What kind of debt do we have most?"

If I run Tyrion in the frontend part of our webapp, I got the following graph where we can find the acessibility type of debt from the above debt comment among other types we use on the project.

An example of the debt graph showing the current distribution of types of debt

You can customize the weight of each type on the global debt score

All kind of debt doesn't have the same impact. A security issue can be more critic than an over complex function. So you may want to weight differently each kind of debt. This can be done by creating a .tyrion-config.json file in the root of your project. You can then assign different value to each type of debt.

Here is the default configuration:

{
  "pricer": {
    "bug": 100,
    "architecture": 100,
    "bugRisk": 5,
    "security": 100,
    "securityRisk": 10,
    "quality": 5,
    "test": 5,
    "doc": 3,
    "ci": 30,
    "deploy": 10,
    "devEnv": 10,
    "outdated": 5
  },
  "standard": 100,
  "ignorePath": [
    "node_modules",
    "README.md"
  ]
}

You can find more information on how to use Tyrion on the README

The trend of the debt may be the most important information

After a while of commenting debt in your code, you will be able to use the second mode that will analyse your Git history and will create a graph of the evolution of your debt. The command to get it is tyrion -e 200. You can replace 200 with the number of days you want to scan backward.

An example of a debt report showing the evolution of the debt

This kind of report will help you and your team respond to the question "Are we creating debt or increasing quality?" It is important to know because if the trend of your debt is increasing, you should invest more in quality. Remember you won't go faster by overlooking quality 😉.

Sharing it with the product/business team

If you are reading this, you are probably a developer, and you probably want to spend more time on quality. You may not be able to do so because someone in the business side of your project prefer that you focus on a short term deadline.

So you need to negotiate with this person. This tool was also build to help you to do so by providing graph and data to support your case.

If you need a csv file to allow business people have a closer look at the data you can generate it with the option `--csv`.

Sharing quality data with product team is a good way to onboard them on the topic

Conclusion

Keeping a high level of code quality on a project for years is something very difficult. One positive side effect of using Tyrion on a project is that developers can grow their quality awareness while coding. If all developers of your team ask them-selves "Is this piece of code I'm writing/reading the best we can do?" for each feature, you will be on a good track to keep the quality high on the long run.

You have a question? Feel free to either contact me:

📧 by email
🐦 on twitter
🐙 by leaving an issue or a pull request on the Tyrion repository
💬 by leaving a comment below

I will be happy to discuss about Tyrion and quality management with you.

Write tests for humans

Mon, 07 Sep 2020 00:00:00 GMT

Let me tell you the story of the day I almost quit testing my code altogether. For the tenth times this week, the CI failed with this cryptic error message "Received value does not match stored snapshot" and a diff that looked something like:

<div
  - className="myClassName" >
  + className="myNewClassName" >
  + <div>
    Hello world
  + </div>
</div>

It felt pointless to fix this test : why does it matter if the class-name is different or I added a div tag? It does not keep me from introducing regressions as nobody in the team reads the snapshots updates.

I was so frustrated, I almost broke the computer, slammed the door to quit my job. But instead, I decided to channel this energy into something more positive. I started thinking about how the project tests got into such a state.

How come everybody agrees that we need to test our code base yet so many developers hate it? Was I the only one being frustrated? (No I was not)

When people say they use TDD, I don’t believe them.
— emily freeman (@editingemily) August 26, 2020

Why is nobody doing something about this?

The Vicious Circle Theory or How you stopped writing test for humans

Here is my theory to make sense of this kind of situation. I have called it "The vicious circle of testing" ™:

A Developer changes the code for a feature. A test fails, but it is not clear why.
The Developer fixes the test. They do not know why the test was here in the first place. They grow used to the idea that it is not worth their time investment to fix it.
Next time, this developer writes code, they spend less time and effort writing tests.
And back to the top...

If this resonates with you, you've been writing tests for the CI to work but not for your teammates to understand your intent. Tests are a collaboration tool, they are meant to be read when they fail by you of your teammates. You need to make sure you can understand them in a month when you have forgotten all about what you were supposed to do.

Tests are meant to be read by humans, not your CI.

How do you write tests for humans, you ask? Here are my 3 tips for you :

Why: You need to know why you are testing. Because testing is a universal truth in software engineering, it is easy to forget why you need to do it on your project. Find what motivates you: it will guide you to write good tests that you want other humans to read.
What: You need to focus. I think it is better to do one thing well rather than try to do it all and do it mediocrely. Choose what you want to test, do not try to test it all.
How: Choose the best tools adapted to your goal. They must help you write quality tests that you can read and maintain

#1: Why do you write tests?

Writing great tests for humans takes time, you need to be invested in what you are doing. You cannot be if you do not know exactly why you need to write great tests.

Do not write tests because you have to. Make sure you and your team know why you write tests.

Automatic testing is often a given when setting up quality tools for your project. Unlike your linter, tests do not fail every day, they are a longer-term investment to code quality. It is easy to lose sight of why you are doing it.

Before starting to set up your test environment you should clarify why you are testing.

What are you hoping to gain in the long term? Better documentation? Fewer regressions bugs? Developer experience tool for TDD?

Make sure you and your team know why you are doing it. Because it is so easy to fall into the vicious circle of bad testing, if you all know why you are driven to write tests, you can catch early when you are deviating from your goal. Making your goal clear will give your teammates ownership over your test and they will help you push for improvements when needed.

#2: Choose what parts need to be tested first

If we chose to use snapshots on my project, it was to go faster. We wanted to test all of the codebase without investing much time. By aiming to keep a 100% coverage, we started writing tests that needed to be maintained every time we added features. They could not help us catch regressions because they changed too often. No matter how small the investment was, it was not worth it.

Do not aim for a 100% code-coverage of generic tests, aim for 100% of readable tests

You do not need to test everything. Your set up may not be adapted to every aspect of your code.

I always write unit tests for very logical functions :

They are hard to read and easy to break. A test is a safeguard to make sure another developer will not break it.
It is cheap to set up and to write.
It is fast so it does not slow down the ci.

But for most part of the code, it is a case-by-case study.

Should I test my frontend app UX? If I am writing a complex single-page app, UX is very likely to evolve. Testing it may be my priority so I can prevent regressions that have a big impact on the user's appreciation of my app. I want to write integration or end-to-end tests for this.

Should I test your frontend app UI? If I am writing a shared-component library, I am focusing on UI and have a few logical components. My first mission is to provide my users with reliable and consistent UI. Testing UX logic is a bonus, it is ok to make choices and not to focus on this at first.

#3 Choose your tools right

There are a lot of great testing tools available. Now that you know why you want to write tests and what parts of code you want to focus on, you can find the tools that fit your needs like a glove

On another project, I chose to test my frontend React app with react-testing-library. I wanted a tool that :

would test our React app components logic and document what they did
would help new developers feel confident when starting on the project
would help us focus on these goals even if the teams changed

That is why I chose react-testing-library to write integrations tests on it. It turned out to be a great developer experience. I also had a personal goal of trying this new tool that had a lot of great reviews.

Find a tool that makes writing test fun, or that makes you want to do test-driven-development, you will only write tests for humans this way.

You are ready!

Now you know: if you ever find yourself thinking testing is useless or painful to maintain, you are probably in a very vicious circle of testing. If you want to escape this, here are the steps to follow:

Remember tests are a documentation and collaboration tool, they should be made to be maintained by teams and read by humans
Make sure your team knows why you are writing tests. This will motivate you to write good tests that you can be proud of.
Do not try to do it all. Coverage is just a number, it is not a token of code quality. Focus on testing some parts well rather than all of it in a poor way.
Choose the tool that fits you and that makes the developer experience fun

Software-based CPU power consumption using PowerApi

Thu, 03 Sep 2020 00:00:00 GMT

The ICT sectors carbon footprint represents now around 4% of the global CO2 emissions. To minimize that, one of our goals at Theodo is to design sustainable web applications with minimal carbon footprint. And in order to know how performant we are at this, we need to be able to evaluate the power consumption of our applications and more specifically the power consumption of the VMs we use in the cloud. Unfortunately, at the moment no cloud provider is able to provide such a data. So we had to look for a solution.

Our research led us to PowerApi, an open-source project developed by the Spirals research group from University of Lille 1 and Inria in France. It allows people to estimate the CPU’s power consumption induced by an application without any external device like a wattmeter. As we can't put wattmeters to measure the VMs consumption in the cloud, this looks like a good option.

The magic formula

From the documentation, we can read that PowerApi collects with a sensor raw data from the hardware of the server. Then PowerApi applies a “formula” and voilà: we have the power consumption of the monitored software stored in a database.

After reading this I asked myself two questions : what is this “raw data” and how did they build this magic formula ? Fortunately, we can all have access to their research papers where they explain everything. So let’s dig a bit deeper.

A Power Model is used to approximate the power consumption

The formula is what is called a Power Model in the academic field. A power model is a model that approximates the power consumption of a hardware based on different factors.

I knew that each CPU has a specific TDP on their specification page: “Thermal Design Power (TDP) represents the average power, in watts, the processor dissipates when operating at Base Frequency with all cores active under an Intel-defined, high-complexity workload”

So when I pictured the formula, I thought it was going to be quite simple and something like:

Consumption = Idle power consumption + %CPU load * (TDP - Idle power consumption)

Reading their research paper, it turns out it’s more complicated: “the CPU load does not accurately reflect the diversity of the CPU activities. In particular, to faithfully capture the power model of a CPU, the types of tasks that are executed by the CPU have to be clearly identified.”

Power consumption is related to the CPU activity which is not exactly the CPU load

The types of tasks in a CPU are also known as events. Each processor type has hundreds of different events with potentially a different power consumption. To have an acute idea of what the CPU is doing and thus knowing the related power consumption, the PowerApi teams decided to rely on those events and count them.

To do that they rely on specific counters of the CPU: the HPCs (hardware performance counters) : “We therefore decide to base our power models on hardware performance counters (HPCs) to collect raw, yet accurate, metrics reflecting the types of operations that are truly executed by the CPU.”

Learning phase: creating the Power Model from the regression analysis of real power consumption

The goal of this phase is to approximate the amount of power consumption per HPC events during a workload. It’s made possible by measuring both real Power consumption of the server and counting HPC events in real time.

Step 1: select an unbiased set of workloads

To ensure that the data is not biased to one kind of CPU activity, the team used an academic set of different workloads called PARSEC.

“To explore the diversity of activities of a CPU, we consider a set of representative benchmark applications covering the features provided by a CPU. In particular, to promote the reproducibility of our results, we favor freely available and widely used benchmark suites, such as PARSEC.”

Step 2: get the correlation between real power consumption and HPC events

The team measured the real power consumption using a bluetooth power meter : the Power Spy 2.

Real time in the context of this experiment is that every 250 ms:

The Power Spy 2 send the power consumption of the last 250 ms to a collector
The number of HPC events of the last 250 ms are sent to a collector

By doing so we obtain a lot of data linking power consumption and HPC events.

Step 3: select the HPC events that are the most correlated to power consumption

The two final steps are a mathematical ones. Step 3 is about removing the HPC events that are weakly correlated with power consumption to simplify the step 4.

Step 4: Create the Power Model using ridge regression

At this step, the team uses an advanced mathematical method of regression (ridge regression), to get the 2 or 3 HPC events that will be able to predict the power consumption of the software on the machine. This is what we call the Power Model

Here is a diagram of the whole phase:

The final Power Model includes memory, disks, motherboard and fans

As the Power Spy 2 measures the power consumption of the whole server, all other hardware components (memory, disks, motherboard and fans, ..) are integrated into the power model. It works well because the CPU is the most energy consumming component.

The final power model is P = Pidle + Pactive, where Pidle corresponds to the power consumption of the CPU when no program is running and Pactive to the additional power consumption from a workload.

For example, one power model for the Intel Xeon W3520 is

With e1 and e2 two HPC events.

To get a real time power consumption of this server PowerApi has then just to count those events.

Real life application

As Cyrielle has explained in this blog post, we were able to measure the power consumption on our local machine. But we couldn't use it to calculate power consumption on AWS servers which was our original goal. Indeed the research team can't do the learning phase on the AWS servers (yet?) as they don't have access to it. So the PowerAPI software can't get the data needed for its Power Model.

Thanks to this project and the things we read to try it, we learnt a lot:

The CPU is the component in a server that consumes the most (~100 Watts). The memory is around 10 Watts for 32GB and a SSD consummes around 2 Watts.
CPUs, when idle still consume around 1/3 of the peak consumption.
The next generation of Intel's CPU may see a 91% reduction in power consumption
ARM processors have a very low energy consumption compared to x86 processor(~2 Watts vs ~100 Watts)
AWS provides arm-based EC2 instances if you look for efficient way of reducing the carbon impact of your web application.

The next idea that looks promising to have a rough idea of the power consumption of our application is the "Cloud Jewels" approach by Etsy adapted for AWS.

Prevent AWS from Reading Your Step Functions Data

Mon, 31 Aug 2020 00:00:00 GMT

AWS Step Functions is the perfect tool to handle long-running, complex or business-critical workflows such as payment or subscription flows. However, a naive implementation could put sensitive data at risk.

What is AWS Step Functions?

AWS Step Functions is an Amazon cloud service designed to create state machines to orchestrate multiple Lambda functions, branching logic and external services in one place. It presents most of the advantages of the serverless services: the scaling is immediate and the pricing is based only on the computing time and the number of step transitions.

The state machines can be defined directly in the AWS Console with a JSON configuration and you directly have a graph representation of your workflow.

AWS Step Functions state diagram

At Theodo, we use Serverless as our framework with the Step Functions plugin to write our AWS Lambda functions, define our resources (DynamoDB and Queues) and setup AWS Step Functions and the state machines directly in our codebase.

Example: an identity verification workflow with Step Functions

The previous picture shows an identification verification workflow. It is a piece of a bigger validation flow for banking applications.

What it does is request an identification URL to a third-party system, send this link by text message to the applicant and wait for them to complete the identity verification. When it is done, the third-party system calls a webhook, which restarts the state machine and either triggers a retry, sends a rejection email, or continues the global validation process.

The identification flow

To perform all of those actions, sensitive information will flow through the inputs and outputs of the state machine steps:

👤 Personal information from the applicant which are sent to the third-party service to check his/her identity
📱 The phone number of the applicant to send the SMS
✉️ The email address of the applicant to send the mail
✔️ The result of the identity verification

Problem: we are exposing sensitive data

With this implementation, we see that sensitive data flow through the inputs and outputs of your state machine directly.

state-machine.yaml
----------

SendIdentificationRequest:
  Type: Task
  Resource: arn:aws:states:::lambda:invoke
  Parameter:
    FunctionName:
      Fn::ImportValue: ${self:provider.stage}-sendSMS-Function-arn
    Payload:
      creditApplicationId.$: $.creditApplicationId
      creditApplicant.$: $.creditApplicant
      phoneNumber.$: $.phoneNumber
      message.$: $.smsBody
  ResultPath: $.identificationSMS
  Next: WaitIdentification

An exemple of sensitive data flowing through our inputs and outputs

The problem is: nowadays, most DevOps engineer, developers and even some product managers with access to AWS and can see that information. Furthermore, AWS Lambda dumps are stored in an S3 bucket which is not encrypted with the default configuration. Any malicious access to your AWS stack could leak sensitive information far more easily than by corrupting your database for example. The more you spread sensitive information between multiple services, the more you put yourself at risk. Finally, as it is stated in AWS documentation:

Any data that you enter into Step Functions or other services might get picked up for inclusion in diagnostic logs.

Depending on your data classification or threat model, this could be totally unaceptable. With that in mind, you may completely dump AWS Step Functions, or you may try to find a solution because the tool is still awesome and perfectly suits your needs!

Our solution

To prevent leaking sensitive information, we need to clean every step function input and output. Here are three ways of doing it:

Configure Step Functions to log in CloudWatch Logs instead and set a low expiry date. This is the easier solution but it still exposes logs for a short duration, and you lose any chance of retrieving old execution informations later.
Keep the data in the input and output but encrypt and decrypt it between each step. This one is a bit more complex but you only need to use an encryption key in your lambdas. However, you are still exposing encrypted data, which can be insufficient for some critical sectors, such as health or defense.
Never transmit sensitive data between steps and retrieve it directly during their execution. This solution requires more services to work and thus more things that could fail. However, it is the more secured and allow permanent recovery of logs.

Second and third solutions also requires break-glass processes to allow developpers to legitimately access log data, for debugging or support matters.

Considering everything, we prefered to implement the last solution because we wanted to keep our sensitive data in a single safe spot, where we put a lot of attention to the security. Also, we were already using secured AWS DynamoDB encrypted with external KMS keys for the waiting tokens of our lambda functions. Thus, we decided to create a new table to store sensitive data between steps.

We configured it like below.

resources.yaml
----------

StepSentiveInputTable:
  Type: AWS::DynamoDB::Table
  DeletetionPolicy: Retain
  Properties:
    TableName: step-sensitive-input-table
    AttributeDefinitions:
      - AttributeName: sensitiveInputId
        AttributeType: S
    KeySchema:
      - AttributeName: sensitiveInputId
        KeyType: HASH
    BillingMode: PAY_PER_REQUEST
    PointInTimeRecoverySpecification:
      PointINTimeRecoveryEnabled: true
    SSESpecification:
      SSEEnabled: true
      SSEType: KMS
      KeyId: hsm-key-id

DynamoDB table definition in Serverless

We were already using DynamoDB from our lambdas to save our waiting task tokens and we chose to add another table to save sensitive data between steps. Furthermore, on AWS, you can choose to save the encryption keys of your DynamoDB tables outside of AWS, which was a big plus for us.

To avoid repeating ourselves, we coded a wrapper that we use on all our lambdas. It does the following things:

After the return of the lambda, the wrapper:

Looks for sensitive data keys in the output, remove them and build an object with that information
Save the sensitive data objects in a dynamo entry
Add the id of the new dynamo entry under a “sensitiveData” key in the output

Before the execution of the main lambda function, the wrapper:

Looks for “sensitiveData” keys in the input
Retrieve the content at the given IDs
Replaces the "sensitiveData" keys with the resulting object in the input

// encryption-wrapper.ts

export default (lambdaHandler: LambdaHandler): LambdaHandler => {
  const decryptHandler = async (event: any) => {
    const decryptedInformation = await getSensitiveData(event.sensitiveInputId);
    return lambdaHandler({ ...event, ...decryptedInformation });
  };

  return async (event: any) => {
    const clearOutput = await decryptHandler(event);
    const sensitiveInputId = await storeSensitiveData(clearOutput);

    for (const sensitiveKey of sensitiveInputKeys) {
      delete clearOutput[sensitiveKey];
    }

    return { sensitiveInputId, ...clearOutput };
  };
};

Our custom encryption wrapper

Last of all, we needed to encrypt the first input at the start of our state machine from the API to finally secure our users information all the way!

Takeways

Take care about what information you have in your lambdas and state machine events on AWS
Always think twice about the data you put in your logs
Spread your secrets and security implementation in as few services as possible
Use a wrapper if you want to repeat behavior in multiple lambda functions.

Thank you very much for reading my article. Feel free to comment, ask for implementation details, and feel free to share the problems that you encountered with AWS Step Functions.

Typescript: use the nullish coalescing operator to prevent bugs

Tue, 04 Aug 2020 00:00:00 GMT

My goal as a CTO is to improve quality. The score of this game is the number of bugs we find each week. Today I share with you a typical bug that more than one person got caught by.

Let's say you want to initialize the audio volume of your react application with the value previously saved in the localStorage or defaulting it to 0.5 if nothing was saved. You could write something like:

Bad example

function initializeAudio() {
  let volume = localStorage.volume || 0.5;

  // ...
}

The problem is that if the user has saved their volume to 0 it will be interpreted as falsy and your code will use the default value instead of the right value. Some devs prefer the simplicity of || than putting an explicit if clause. And more than once, they were right because 0 was not a plausible value. But as a standard it is too dangerous. For example someone else could see the code and think the || is a good coding stantard in all situation which will eventually create a bug.

Typescript 3.7 comes with the best option: you can now use the nullish coalescing operator to prevent that wrong behavior and safely write something like:

Good example

function initializeAudio() {
  let volume = localStorage.volume ?? 0.5;

  // ...
}

That will be compiled in:

function initializeAudio() {
    var _a;
    var volume = (_a = localStorage.volume) !== null && _a !== void 0 ? _a : 0.5;
    // ...
}

To make sure people from your team use it, you can use the prefer-nullish-coalescing ESLint rule. You can also use it with javascript and this babel plugin. If you want to go deeper in the understanding of this operator you can go here.

I hope this article will help your team prevent this kind of bugs. For more actionable code quality tips, find me on Twitter.

Implement contract testing with Pact to get rid of bugs due to front-back and back-back communication

Sun, 05 Jul 2020 00:00:00 GMT

Image from the Pact documentation

Exec summary

Modern software development organizations scale their development efforts by spreading the development of a system across different teams. Such projects can have team work issues because interconnected applications that change often, create a lot of errors breaking other teams' application. A typical problem due to a large organization is when the backend team deploys a change that is not compatible with the mobile app and breaks it.

Contract testing is a way to make sure that several applications developed by collaborating teams can understand each other. With no need to take extra manual care when modifying code, the teams avoid human errors, get rid of bugs, and on the way, develop faster.

With contract testing, a team can easily create a test checking that the application they are developing receives viable responses from the provider application that another team is developing. On their side, the provider application team only has to run a single automatically generated test. This allows both teams to know if something is broken before they deploy their applications: they can talk quickly about this precise problem, with little rework cost and no bug for the users.

Pact is the main framework to implement contract tests, and it works with many languages.

For Whom?

In order to avoid creating bugs coming from miscommunication between two parts of your applications you can set up different tactics, contract testing is one of them. Here are some alternatives with an evaluation of how they perform. You can find some comments on these alternatives in the appendix at the end of the article.

Alternative	Maintenance cost	Setup cost	New interaction cost	Learning curve	Efficient
End-to-end testing	$$$	$$$	$$	$$	++
Shared types	$	$$	$	$	++
Versioned API without breaking change	$$$	$$	$	$$	+++
Massive documentation and conception	$$$	$	$$$	$$$	+++
Contract testing	$	$$	$	$$$	+++

All these tactics can be used together where it is possible.

In fact contract testing needs some conception to create the contract or update it. But contract testing allows to have an accurate documentation when you want to do this conception. Once you discover a breaking change you still have to take a decision on how to handle it and versioned API are a good way to do it but the contracts will show you what you do not need to maintain because no one uses it anymore.

Conclusion: if you develop a distributed system communicating via HTTP APIs with heterogeneous technologies you should find someone to help your team set up contract testing.

Benefits

Benefit #1: Defects are stopped before wasting others time

Usually to achieve testing of several services interacting, we use some kind of end-to-end testing. This means you can test only after deploying your service to put it on the same environment as other services (e.g. staging, preprod, CI). What is called end-to-end testing here could be called broad stack tests, it matches the definition found here. If there are several teams, it becomes difficult to have someone responsible for end-to-end tests.

With contract testing you can test directly in your continuous integration (CI) workflow which means no one else than the developer invested time in checking the work done, this allows problems to be caught earlier. (cf. process just below)

doing > manual local test (few minutes) > push (few seconds) > CI (few minutes) > code review (some minutes) > merge (few seconds) > deploy (few minutes) > manual test on staging > validation by the PO

Benefit #2: Developers know their mistakes and correct them quickly

As you can see on the process above having the result in CI instead of after deployment creates a faster feedback loop for the developer and he can correct it before the merge which means corrections are done faster also.

Let me introduce the pyramid of tests briefly, there are several kinds of tests with different drawbacks, the idea is that even if useful, the more drawback you have, the less tests of this kind you do.

Image from the Martin Fowler article Test Pyramid

The main advantage is that contract tests are lower on the pyramid of tests (between unit for the cost and service for the speed) than end-to-end tests (UI). It means they are faster and cheaper, so we can test more cases which gives quickly more confidence for less money.

Benefit #3: Contract tests are easy to create and maintain

With the pyramid we saw that contract testing is cheap compare to the end-to-end alternative. Our experience is that once set up the cost is close to unit tests (only slower) which are widely used on most projects.

Each test, tests exactly one case, this means they do not change often (compared to end-to-end tests where any change can break the test) and that a change is easy to do in the test because the complexity is low (like a unit test).

Benefit #4: Contract tests give auto-documented API usage

Thanks to Pact that registers all your contract tests, you know precisely what are the interactions tested in your project. This is up to date even before going to production and you can easily visualize it from Pact.

Services interactions:

Image from the Pact documentation

Details of an interaction:

Image from the Pact documentation

Trade-offs

Trade-off #1: It is useful only when several teams work together

Tests are used to perform check that are hard to remember and take a lot of time to perform manually. For contract tests this means they are less useful when you only have one or two teams as you usually know exactly what the other team uses.

They remain relevant but the cost for an inexperienced team on how to use them could create more problems than it solves at first. For an experienced team the cost is not an issue making them a practical solution even to test the interactions between one frontend and its API developed by the same team.

Trade-off #2: The setup is quite complex at first

To make contract tests work you need several days of work.

All teams need to understand the core concepts and how to write good tests with the good level of testing.
You need to add a pact broker in you environment (meaning you have to deal with security issues and keeping it running) this is mitigated by the fact the provided docker container works fine.
You need to make sure your providers applications run in your CI. By default there is no need to have a running instance in your CI but this is quite easy to do with docker.
You need to find a way to handle state in your provider apps to simulate all kinds of responses for instance you need to simulate your own dependencies to make the test reliable. Depending on the complexity of your dependencies (number of dependencies and number of possible outputs) this can be more or less difficult.

My Opinion

You can benefit of contract testing in many situations. Once you have overcome the initial cost to set up and understand how you should work with it for a big project it can even be a cost effective solution for smaller projects in your organization. So join the Pact users' community to improve your test stack and help your teams to ship faster. You are not the first, many companies like Atlassian use this tool already as they explain it in this video.

Go Further

To know if you can get benefit from Pact you should read this page of the documentation, in case of doubt read this. Then to learn how to use it you should begin here and here. To know where these tests should be used you can read the excellent article The Practical Test Pyramid which describes most kinds of test you can use in a project.

Appendix

If you want more details, here are some elements I used to create the evaluation table of the alternatives in the "For Whom?" part:

End-to-end testing

Pros:

When the tests is green you are sure everything works together since the tests checks everything at once

Cons:

The cost of maintaining an environment to test everything.
Test tend to be very flaky due to external causes.
You need to run all tests (and they are slow) before the real deployment of any part, which creates a congestion in your teams workflow.
The coverage of error cases costs even more because you need to set up your system to fail in expected ways and because each case is quite long to execute you tend to have few.
When you discover a real bug, you still need to work to find the root cause (slow to fix)

Use: When you want to check your most critic happy path and you have a way to have enough control on your system in an environment before prod (set up mocks for external services, set up well known data...)

Shared types

Pros:

Easy to set up

Cons:

You do not have the vision of what versions work together, this force to deploy the different pieces together which is a smell for down time.

Use: When you have a small enough system that you can deploy all at once and a technical way to share code between consumers and provider.

Versioned API without breaking change

Pros:

The problem cannot exist

Cons:

Maintenance cost is high, you either have to live with the burden of all your previous choices or find a way to know what is not used anymore (one way is a deprecation plan of older versions).
Risk of human error and still create a bug

Use: For public APIs this is a good solution as there is no way to contact each consumer directly.

Massive documentation and conception

this kind of massive

Pros:

Reduces the amount of bugs of any type.

Cons:

Forces you to make the perfect conception, keep track of every decision, to have comprehensive checklists and to train your whole team to follow all items.
Your process needs to be at the highest level, like the NASA for space shuttles.

Use: When lives depends on your software working perfectly

Contract testing

Pros:

Catch bugs between teams before deployment
Easy to maintain
Teams can deploy at their own pace
Documents expectations of all consumers and ensure they are met with no human intervention
Keeps track of what versions are compatible

Cons:

You need to carefully set up your workflow to integrate these tests.
You need to create tools to handle complex cases.
There is yet another tool to teach to your team.
You still need other kinds of tests to check the business rules of each application work as expected

Use: When you want to quickly develop a system with more than two interconnected applications (if you do not control the update of some applications like for mobile, each used versions count). And each application is developed by independent teams which communicate frequently. It can be better to have someone to train your team to reduce the setup cost.

Serverless Is More Than a Trend, Best Companies Adopt It

Mon, 29 Jun 2020 00:00:00 GMT

A recent article by the WSJ highlighting the benefits of serverless for organizations points at a building awareness by thought leaders about the impact this new trend can have for companies. It resonates with our own experience, celebrating successful serverless implementations with our clients in the last couple of years.

Yet, when pitching prospects about the value they can unlock through this technology, we often find that serverless is not very well known among decision-makers.

If the value is clear, why is there a lack of awareness?

The Term Serverless is Confusing

Ben Ellerby defended the serverless concept in an article, noting that it “is not a good term, yet it is used to describe a powerful and often misunderstood concept.” So, here is his definition:

When we first introduce people to serverless, there is this aha! moment when they understand it doesn’t mean there are no servers -- it’s just that they’re further outsourced to cloud providers.

This graphic positions serverless in the cloud landscape:

In a nutshell, using serverless gives you the ability to develop your own applications and microservices, to integrate off-the-shelf services like payment or user authentication, and to process events generated by these services or other SaaS. Meanwhile, the size of the engineering team decreases, as the full responsibility of managing the infrastructure is delegated to a cloud provider.

Serverless, Right for Large Companies as Well as Start-Ups

Gartner already mentioned serverless computing as one of the top 10 trends in 2019. Allied Market Research expects the global serverless architecture market to reach almost $22 billion in 2025, from $3 billion in 2017. Led by IT & Telecom, they expect the revenue to increase in all sectors, with the highest growth in the Media & Entertainment industry.

Large companies drove the market at its beginnings, with use cases such as real-time web applications and data processing; but SMEs will exhibit the highest growth rate between 2018 and 2025, according to the same study.

In his article Serverless is for Everyone, Sheen Brisals, Senior Engineering Manager at The LEGO Group, explains that “for a majority of start-ups, the tech stack is something they are born with”, therefore “change is easier when compared with a big enterprise”. But “starting small with serverless is perfectly applicable to a start-up as well as to a grown-up.”

As cloud providers invest in improving their solutions, more developers adopt serverless for developing microservices and functions, expanding the potential use cases.

Download the free presentation here

It’s Not Just About Saving Costs. Serverless Computing Solves Business Problems and Delivers Better Customer Experiences.

Reduced operational and development costs are the most important benefits sought by organizations going serverless, together with the automatic scaling of servers according to demand. On the product side, the increase in developer productivity and reduced engineering lead-time has a positive impact on the speed of delivery, thus on the time-to-market.

To overcome the challenges of dealing with legacy applications built during their 100+-year existence, companies like Liberty Mutual embraced serverless computing to solve real customers’ problems. Among the solutions that the IT team have built, Gillian Armstrong (Solutions Architect at Liberty Information Technologies) and Mark McCann (Architect) mention:

A virtual agent for their call center to answer the most common questions.
A virtual assistant to help the clients manage their policy documents.

According to Armstrong, with serverless:

The cost is so low, because it's all managed services, and they're able to bring it out, trial it with users for a little bit, make sure the whole system works, and then just scale seamlessly up.

At Theodo, we're also obsessed with business outcomes, and we train our engineers to focus on delivering value for our clients' end-users, in line with our Lean culture. When we start a new project, we recommend our clients use serverless components when we're confident it will add a competitive edge to the solution.

Check in this video to see how our ground transportation client used serverless for managing omnichannel, real-time notifications to passengers:

Serverless Computing Is as Sophisticated as It Is Powerful: the Main Roadblock is Training

According to the O’Reilly Serverless Survey 2019, with a panel of 1500 respondents from a wide range of locations, companies, and industries, some of the biggest challenges are linked to the education and hiring of staff.

In the organizations that have not (yet) adopted serverless, “fear of the unknown” comes in second after security concerns.

Some components proposed by Cloud providers, like DynamoDB from AWS, have huge potential, but using them correctly is still a skill that requires mastery.

To tackle these concerns, Theodo developed EventBridge Storming workshops, adapted from the Event Storming workshop technique by Alberto Brandolini. During these sessions, we work with business stakeholders to define a common language around application fields (e.g. order, payment, and shipping) that we can use to communicate with the team. We then use this vocabulary to design an architecture based on events grouped by domain and well-defined system boundaries.

Cloud providers and the user community also recognize the challenges faced in upskilling teams to handle serverless. The good news is that they are doing an outstanding job at improving the tools, developing best practices around security concerns, and sharing knowledge on novel ways to solve issues. Theodo is proud to have contributed to multiple frameworks and libraries in the space, an achievement recognized with a partnership with AWS and the Serverless Framework.

Conclusion

Reductions in Total Cost of Ownership and Time-to-Market are two compelling reasons to consider serverless for IT leaders. Besides, the granular nature of FaaS and managed services allow companies of all sizes to progressively build new applications and replace pieces of legacy architecture without requiring an engineering big-bang.

Chances are that you're already using serverless. If your technical stack includes a cloud provider such as AWS, Azure, or Google Cloud, your engineers have probably used their FaaS offerings for data processing or queue services for notifications. You're not starting from scratch!

This means that serverless is one of the easiest technologies to adopt. Even if your main infrastructure is still bare metal, you can experiment and iterate with serverless on a limited scope quickly and seamlessly. It will give you a better idea of the benefits and challenges of serverless for your company, right now.

Need help? Contact me at alexdb@theodo.com

Cover photo by Danielle MacInnes on Unsplash

Build an Animated Accordion List in React Native

Sat, 27 Jun 2020 00:00:00 GMT

Accordion lists (or ExpandableListViews) are now a material design staple and show up in quite a few apps. They are a great way to dynamically display information to a user.

When it comes to implementing one in your own app, sure, there are libraries that you can use. But implementing one yourself gives you greater control over styling, a smaller app bundle size, and it’s so easy it might even be faster! It’s also a great opportunity to sharpen your skills using React Native Hooks and the Animation API.

This is what we’ll be creating:

Just show me the code!

Alright, alright! Here’s a snack. Consider sticking around though - you might learn something.

Let’s get started

First off we need a functional component. This will take two arguments or props:

A string. The title in the top bar of the list item.
A ReactNode. The component in the expandable section.

For the second of these, React allows us to access child components via this.props.children in class components, or we can specify children as an argument to our functional component. Magic!

So our functional component has the form:

const AccordionListItem = ({ title, children }) => {
 ...
};
export default AccordionListItem;

And we can use it like so:

<AccordionListItem title={"List Item"}>
  <Text>Some body text!</Text>
</AccordionListItem>

The top bar of our list item will need to be wrapped in a TouchableWithoutFeedback since we want users to be able to open and close the list item by pressing it. This can contain the title we passed in via props.

<TouchableWithoutFeedback>
  <View style={styles.titleBackground}>
    <Text>{title}</Text>
  </View>
</TouchableWithoutFeedback>

Now let’s add a View for the list item body. This is the section that slides in and out when the header is clicked, and will contain our child component.

<View style={styles.bodyBackground}>
  {children}
</View>

Let’s see what we have so far:

Nothing revolutionary right? Bear with me, here’s where it gets interesting!

Now I’m hooked

Next up we’ll want to add a few React hooks to keep track of the state of our component. Here they are:

A state to keep track of whether the item is open or closed.
An Animated.Value to animate the opening and closing of the body section.
The height of the body section - this is what we will be changing in order to show and hide it.

Or in code form:

const [open, setOpen] = useState(false);
const animatedController = useRef(new Animated.Value(0)).current;
const [bodySectionHeight, setBodySectionHeight] = useState(0);

Note that we’re using useRef to create the animation object. Since our open state will persist across component refreshes, we want our animated value to do the same, otherwise they’ll get out of sync! useRef acts as a sort of state container for our Animated.Value, elevating it out of the component lifecycle.

Accordion time!

Now we need to tie our Animated.Value to the height of the body view. Our animated value can vary over whatever range we want but it’s easiest to pick a simple range like [0, 1]. Here, 0 maps to the list item being closed and 1 to it being open. As the animated value changes from 0 -> 1 we want the height of our view to go from 0 -> bodySectionHeight. To achieve this we can use the interpolate function:

const bodyHeight = animatedController.interpolate({
  inputRange: [0, 1],
  outputRange: [0, size.height],
});

Which simply maps the inputRange to the outputRange exactly as we described. When we change the value of the animatedController, the bodyHeight variable will scale this value relative to the height of our container. With all this set up, we can now write our toggleListItem() function, which will be called when we click on the item header.

We can think about what this function has to do as follows:

If the list item is open:
a. Change the animated value from 1 to 0.
b. Set the state to closed.
If the list item is closed:
a. Change the animated value from 0 to 1.
b. Set the state to open.

const toggleListItem = () => {
  if (open) {
    Animated.timing(animatedController, {
      duration: 300,
      toValue: 0,
    }).start();
  } else {
    Animated.timing(animatedController, {
      duration: 300,
      toValue: 1,
    }).start();
  }
  setOpen(!open);
};

I’ve set the duration to 300ms, but you can play around to find your preferred speed.

We’re getting pretty close. We’ll need to make our View an Animated.View in order to use React Native’s animation system, and then set the height of the animated view to be the bodyHeight variable.

<Animated.View style={[styles.bodyBackground, { height: bodyHeight }]}>
  {children}
</Animated.View>

Now for a sneaky step. We need to set an initial value for the height of our bodyView so we know what value to return when we open the list item. But, we can’t use the height of the animated view. This is because we are going to set the bodyHeight using the onLayout prop of View. For an animated view this is called many times while the view is animating! This is no good for us, we just want to set it once.

To fix this we can nest another regular view in our animated view which can have a constant height. We can then use this view’s height for the value of bodyHeight.

onLayout takes a function with an event parameter. We access the components height via event.nativeEvent.layout.height. So, putting it all together:

<Animated.View style={[styles.bodyBackground, { height: bodyHeight }]}>
  <View onLayout={setSize} style={styles.bodyContainer}>
    {children}
  </View>
</Animated.View>

Finally we need to set the overflow: “hidden” property on the bodyBackground style. This tells the children of the animated view what to do if they overflow the views box, which is exactly what will happen when the list item is closed. Setting this to hidden means they won’t be visible, perfect!

Realistic slide

What we have is nice, but it would be great if we could give the impression that the body of the list item is unfolding from the title bar. Fortunately there’s an easy way to achieve this! We can add the following properties to the bodyContainer style:

position: "absolute";
bottom: 0;

This ensures that the bottom of the inner view is always aligned with the bottom of the outer view, meaning the inner view should slide nicely up and down as the list item opens and closes.

Spinning arrows

Another nice feature would be to have an arrow indicator that rotates as the list item opens and closes. With our new knowledge of React Native animations, this should be easy! As our animated controller varies from 0 -> 1, we want the angle of our arrow to vary from 0 -> π radians or 0 -> 180° for regular folks. Using the same animated controller means it will always be in sync with the opening and closing. So we have:

const arrowAngle = animatedController.interpolate({
  inputRange: [0, 1],
  outputRange: ["0rad", `${Math.PI}rad`],
});

Then we can add our chosen arrow icon, within an Animated.View, into our title container view:

<View style={styles.titleBackground}>
  <View style={styles.titleContainer}>
    <Text style={styles.titleText}>{title}</Text>
    <Animated.View style={{ transform: [{ rotateZ: arrowScale }] }}>
      <DownArrowSVG width={20} height={28} />
    </Animated.View>
</View>

Here we have used the transform CSS property to pass in a z-angle of rotation for the view (in other words, the angle perpendicular to the z-axis - I had to work this out by trying rotateY and rotateX first!).

Take it Easing

One for the true perfectionists out there. React Native allows us to change the velocity of the animation via the easing parameter. This is a subtle change in our case, but it’s a good habit to get into as it can make a big difference in feel for some animations.

Most of the time I use the standard Material Design easing:

easing: Easing.bezier(0.4, 0.0, 0.2, 1);

Putting this in both our Animated.timing configs, we get our finished product!

Even after all that work our list item is less than 1kb in size once gzipped! I challenge you to find a library that’s smaller. We also have complete control over every aspect of our styling. Stick this in a FlatList component to create a nice scrollable list, or you can just stack them like regular components if you only need a few. Have fun with it!

Why it’s time to consider Azure to host your static application

Mon, 15 Jun 2020 00:00:00 GMT

The line between static & dynamic web applications is not what it used to be. With new frameworks like Gatsby or NextJS, the Jamstack is a strong option to get your application running in no time, whilst being able to scale it as needed.

When it comes to hosting your app, there are dozens of services available today, from Netlify to Firebase, but Microsoft has never really been one of those options (apart from hosting your app using an Azure Blob Storage) until last month, when they released their new service in preview, Azure Static Web Apps.

Here are the reasons why I think it is worth considering.

Get your app hosted & running in no time

If you haven’t used Azure before, you’d be surprised how easy it is to get started, and not only will you get set up in no time, but you’ll benefit from 12 months of Free Tier on most used services. Check it out here

Once you’ve picked your favourite frontend boilerplate and pushed it to Github, the only thing left to do is specify in your Azure Portal which branch you want to use. In less than a minute, your app will be hosted and available publicly.

In the background, Azure Static Web Apps service will generate & push a Github Action template to your repository, which will be triggered every time a branch is updated, creating for you multiple environments (even for each PR) without having to worry about CD. You can then enhance this action to add your own unit and integration tests.

Take advantage of the Microsoft Developer Experience

We have to admit that Microsoft have done an amazing job with VSCode. It is today, without any doubt, the most used IDE for web developers (haven’t tested it yet? check it out here https://code.visualstudio.com/ - In Theodo UK, we have decided to make it our default environment for web projects) And with such tools, the more popular they are, the more their ecosystem will become rich thanks to its community.

In our static web application example, it takes 2 clicks to download a LiveServer extension and have a server running your code locally, with hot reload. If you’re then planning to retrieve data in your app from an API, check out their extension which allows you to simulate your Azure Function running locally, test it, and deploy it all of that from your IDE.

Finally, you can write your app in TypeScript, framework designed by Microsoft, to type your JS code on your frontend and API, for a smooth and reliable experience.

Rely on Serverless to scale your app

If you’re not familiar with Serverless, we could summarise it this way: it is the concept of not provisioning servers, relying only on cloud services, to host your code and scale it up and down as needed automatically, greatly reducing the maintenance cost. (want to learn more? check out https://serverless-transformation.com)

As I mentioned just before, Azure Static Web Apps service allows you to use an API to retrieve data in your app. To do so, Microsoft give us the possibility to use Azure Functions, in JavaScript or TypeScript, to then be called directly from your frontend code. Once your environment variables defined, you just have to configure your SSL, your domain name and your app is good to scale!

Conclusion

Since Microsoft have gone down their open source journey, they are becoming a strong contender for a move to the cloud. Want to get started? Check out their quickstart

Don't overestimate the importance of performance when choosing a stack for your Web project

Thu, 14 May 2020 00:00:00 GMT

Understand how a suitable backend language can make a server handle higher loads and help reduce operating costs – or not.

A few months ago, a colleague of mine gave a talk detailing how great his recent experience with Go had been. As a side-note, he mentioned that Web applications written in Go were able to handle way greater loads than those powered by its mainstream Web competitors. This intrigued me: if this performance gap does exist, what is it due to, and is it significant enough for me, a Web developer, to actually care about it?

After a bit of research, I was confident I had a better understanding of the factors at play in the performance of Web servers. Still, I was lacking some quantitative insight that would allow me, at last, to decide whether taking these differences into account would help me bring more value to my next project. This is when I chose to run a small home-made benchmark comparing Node, PHP, and Go – three technologies we use at Theodo – that I will use to draw the final conclusions of this post.

What is performance anyway?

The first step of my investigation was to get sure of what I meant by performance. Although there is no single definition of this concept, I am confident that the following is relevant to my needs as a Web developer:

Performance is the ability of a server to handle a great number of requests during a given period of time.

Thus, a server that performs well according to this definition will be able to run on less powerful hardware, reducing operating costs and environmental footprint.

It is also correlated to other definitions of performance you may have thought about. For example, the faster a server can respond to a request, the more requests per minute it should be able to handle.

What a performant server is made of

The next thing to realize when trying to understand how a particular technology may influence the performance of a server is that during the whole process of serving an HTTP request, time dedicated by the CPU to executing code actually written by the Web developer or by the Web framework developers is often extremely small compared to:

Time spent performing heavy computations, such as hashing a password, manipulating images, etc.
Time spent waiting for network calls to terminate. Our server is most likely communicating with distant APIs and databases, or with the local file system. This takes way more time than executing a few thousand instructions on a CPU.

As counterintuitive as it may seem at first, heavy computations will not make much of a difference here. "Slower” interpreted languages will typically call well-optimized binaries under the hood to handle those kinds of tasks.

[Summary]:
   ticks  total  nonlib   name
      0    0.0%    0.0%  JavaScript
     72   63.7%  100.0%  C++
      1    0.9%    1.4%  GC
     41   36.3%          Shared libraries

What will make a difference on the other hand is the way the server uses the waiting times during the processing of a request to start handling other requests concurrently.

Scheduling how the CPU will handle numerous concurrent connections to the server is a complicated problem that can be addressed in a number of ways:

Typical PHP servers will let OS threads handle the task. To put it simply, threads are the mechanism that operating systems use to give us the feeling that they are processing several tasks in parallel. Thus, threads are also natural candidates for managing the concurrent processing of HTTP requests – each thread being responsible for a single connection – even though this is not what they were primarily designed for. This unfortunately comes with some performance issues: each new thread consumes quite a lot of memory, and time spent switching from the execution of a thread to another is also non-neglectable.
Technologies such as Node or Go will keep the processing of all the requests in a single or few threads but will use their own user-land mechanisms, such as an event-loop (Node) or Goroutines (Go), to address the scheduling issue. Although these two mechanisms will make Web developers write very different-looking code, they have in common that they do not suffer from the same overhead as OS threads do when opening a new connection or when switching from a connection to another.

Benchmark

We now know that thread-based Web servers should in theory be outshined by their counterparts that use a more suitable concurrency model. But is the performance gap wide enough to be of any importance to developers? Let’s find out!

To carry my little experiment, I set up a server and wrote three different routes in PHP, Nodejs, and Go (more details about versions and my exact protocol in appendix):

When route 1 is fetched, the server hashes a string then returns OK.
When route 2 is fetched, the server performs a call to a (very) distant server, waits for the response, then returns OK.
Route 3 is a mix between route 1 and route 2: when it is fetched, there is a 1% chance that the server hashes a string, else it performs a call to the distant server. It emulates a server that performs a lot of I/O (database queries for instance), and some CPU-intensive tasks (such as hashing passwords to identify users).

To compare performance, we’ll put some load on every route, one by one, for every language, and use the following comparison criterion:

Reference load = the maximum number of HTTP requests a server can handle in a minute, such as 95% of requests take less than 3 seconds.

Route 1 falls into the heavy computations category, so we expect to see little difference between all three technologies:

The small variations we observe can probably be explained by slight differences in bcrypt implementations. An interesting fact to notice here is that the reference load could have been predicted by dividing 60 seconds (1 minute) by the time it takes to complete a single isolated request (PHP case: 60 / 0.320 = 188). This means that our server is simply processing the requests one after the other, and has no opportunity to parallelize anything since there is no waiting time while handling a request.

So far, our results match the theory. Great!

Route 2 is much more interesting since there is now plenty of time for the server to parallelize requests while it is waiting for an API call to return. Let’s see what happens in this case:

We notice that the time it takes to handle a single isolated request is exactly the same no matter the language, which seems natural since the bottleneck of our route is a very long network call. However, huge differences arise when we start putting some load on the server:

Our Node server can handle about 20% more load than its PHP counterpart, which is significant, but definitely not as spectacular as the 1-to-4 ratio in RAM consumption. This seems to validate that thread-based servers are more memory-intensive than servers that use a more specialized concurrency model!
Meanwhile, our Go server is able to keep a low memory footprint while handling almost twice as many requests per minute than the former two. I’ll admit that I have yet to find a satisfactory explanation for this result, and if you do have one, I’d be very grateful if you could take the time to share it with me!

However, a real-world server could very well need to perform a few CPU-intensive operations among all these network calls, which is what I tried to emulate with route 3:

The most noticeable thing, in this case, is PHP’s memory consumption, and it is yet far from reaching any critical level. Reference loads on the other hand seem once again fairly similar.

Let's talk about money

All that is very interesting, but what matters in the end is how much money you will have to spend to run your application.

As an example, let's consider a large Symfony (PHP) project we are currently developing at Theodo. It handles about 50 million requests a day and runs on 32 machines with 8 cores and 32Gbytes of memory each. A similar infrastructure on AWS roughly costs $100,000 a year.

Our benchmark says that a more performant language could potentially divide infrastructure requirements by two. This would mean a $50,000 economy per year.

Conclusion

These results are telling me that choosing a suitable technology could have a measurable impact on operating costs if my service is mainly performing I/O, with close to zero CPU-intensive tasks. Otherwise, my application’s bottleneck will lie in the CPU-intensive code, and the technology I choose for writing my controllers will be of no importance whatsoever.

Let's also keep in mind that there are many more practical considerations that should be taken into account first when choosing a production-grade technology, as they may have a more significant impact on a project success. For example, a rich and mature ecosystem should come with a greater development speed, and should make it easier to find experienced developers to hire.

Note that Web performance in general should definitely be a concern for whoever wants to build a successful product! For that, you can check our performance experts at Theodo [FR].

Appendix: more details about the benchmark

The benchmark was run on an AWS EC2 t2 micro, Ubuntu 18.04.

Nginx 1.14.0, PHP-FPM 7.2
Node 12.14.0, managed by pm2 4.2.1 (and Nginx as a reverse proxy)
Go 1.13.4 (and Nginx as a reverse proxy)
I used loader.io as a load generator.
Code available here.

Measure the server-side impact of your application with PowerAPI

Thu, 14 May 2020 00:00:00 GMT

According to a report made by The Shift Project, the carbon footprint of the digital is estimated to exceed air travel with more than 4% of the total greenhouse gas emissions. This number is expected to double in the next five years!

Green IT principles aims to reduce the impact of the digital. As a developer, I wondered how these principles could be used for the web applications I worked on, and what could be improved.

I first looked at the definition of digital. It includes:

Networks (for instance wifi and 4G)
Data centers (used to host the servers and the data)
Terminals (smartphones, computers, TVs)
IoT (connected objects like a home assistant or a smart thermostat)

In this chart, the Shift Project shows that the energy consumption can be divided in two factors:

45% of the total emissions is dedicated to the production of digital equipment. To reduce the carbon footprint, it is important to keep a computer or a smartphone as long as possible. However, the average lifespan of a computer is estimated around 2 years. As developers, we can have an influence on the lifespan of our users' devices by making sure our applications are not too slow on old devices. Indeed, when you need 30 seconds to open most websites, your first thought usually is: "I really need to change my phone"!
55% of the emissions come from the usage of the equipment. We can have an impact on this energy consumption by designing sober and low-consuming applications.

So how can we measure the impact of a web application on our user’s device?

Some websites like Website Carbon ou EcoIndex (for french users) can estimate the client-side carbon footprint of a web application.
There is also the 1-byte-model which estimates the global carbon footprint. It takes into account the energy consumption of the data center, the network and the user device. A browser extension exists to see the impact of a website.

The data centers represents 19% of the digital energy consumption and some improvements in efficiency and performance can be found. However, there is no easy way to measure the carbon footprint of the server-side of an application. Therefore, it is complex to know where to begin to reduce the impact!

Example of a web application

At Theodo, we use an internal application to display the developers' current missions and availabilities. It is used by the sale representatives to see which developers are available and to staff them on new projects. One person has access to a CSV file online, and updates it several times a day. It is important to always have up-to-date data on the application in order to correctly staff the developers.

To update the application database, a script runs every 15 minutes: it imports the CSV file and parses it, then modifies the availability of the developers in the database. The file, which is quite large, is therefore imported 96 times a day. I had the intuition that it was consuming a lot of energy to run this script as many times, and I looked for a way to confirm or deny it.

I searched “How can I measure my server energy consumption” online and found out that simplest way was to plug a wattmeter into my computer, run the script locally and monitor the energy consumption. However, like most people, I don’t have a wattmeter at home, and I was not very keen on buying one for this unique usage.

I finally discovered a very useful tool: PowerAPI.

PowerAPI

PowerAPI is an open-source project developed by the Spirals research group (University of Lille 1 and Inria) in order to estimate the power consumption of a software, without any external device like a wattmeter. The calculation is made based on the analysis of the consumption of various hardware components (for instance CPU, memory, disk…). The objective is simple: design less energy-consuming software by easily monitoring their power consumption.

The power meter created by PowerAPI is composed of two components: the sensor and the formula

the sensor collects raw data from the hardware while the software is executed and stores it in a database.
the formula accesses this database, processes the raw data into a power consumption in Watt. The formula can store the results in the database or export it in a CSV.

As the sensor needs information from the Linux kernel, PowerAPI can only be used on Linux machines and not on a virtual machine.

Install PowerAPI

To install PowerAPI, you need:

Docker installed on your machine
a MongoDB database that can be accessed by your machine.

In my case, I decided to execute the project locally and deploy the MongoDB instance on my computer.

In the MongoDB database, I created a database (I decided to call it power-data) and two collections:

a collection to store the raw data from the sensor (in my example sensor-data)
a collection to store the computed results from the formula (formula-data)

You can either monitor the global power consumption of your machine, or monitor the power consumption of all Docker containers. As my project was using Vagrant, I decided to monitor the global power consumption of my computer. However, I recommend monitoring the Docker containers as it will reduce the “noise” created by your machine.

To install PowerAPI, the PowerAPI documentation will guide you. Once the MongoDB instance is created, the power meter can be deployed in two commands and a few minutes. Here’s a rapid recap, to measure the global power consumption:

Deploy the sensor: launch the first command to create the sensor Docker container.

docker run --privileged --net=host --name powerapi-sensor --privileged -td \
       -v /sys:/sys \
               -v /var/lib/docker/containers:/var/lib/docker/containers:ro \              
               -v /tmp/powerapi-sensor-reporting:/reporting powerapi/hwpc-sensor \        
               -n powerapi-sensor \
               -r "mongodb" -U "mongodb://ADDR" -D "power-data" -C "sensor-data" \
               -s "rapl" -o -e RAPL_ENERGY_PKG

Deploy the formula: launch the second command to create the formula Docker container.

docker run -td --net=host --name powerapi-formula powerapi/rapl-formula \
       -s \
       --input mongodb -u mongodb://ADDR -d power-data -c sensor-data \
       --output mongodb -u mongodb://ADDR -d power-data -c formula-data

The power meter is now ready, with two running Docker containers: the sensor and the formula.

Start measuring

If both Dockers are running, the power estimation is stored in the database. To stop monitoring, stop the Dockers.

The results

Once PowerAPI was deployed:

Step 1: I first started to monitor my computer energy consumption, to estimate what was the “noise” created by my machine.
Step 2: I launched the vagrant to start my application
Step 3: I ran the script 5 times in order to get average data.

The formula had stored all my power consumption data in the database. Now, I needed a way to display and analyse it.

How to process data from PowerAPI

First, I decided to use MongoDB Charts to display my results.

MongoDB provided me a simple interface to display the results written by the formula. I can for instance filter by date, to create a graph for a precise duration. On this chart, I displayed the energy consumption of my application during 1 minute.

However, in order to extract more information from the results, I decided to extract the data as a CSV, which can be done:

directly in the formula parameters by adding --output csv at the end of the command to launch the formula Docker
with a MongoDB command
via MongoDB Charts.

Once I had the data, I started to analyze it.

The results were given in Watts. When dealing with electric consumption, it is easier to use Watt-hours: for instance, you need 1 kWh to bake a cake in an oven during 1 hour. It means that your oven consumed 1000 Watts during 1 hour.

	Duration (s)	Average Power (W)	Consumption (Wh)
Step 1	225.5	0.81	0.05
Step 2	225.5	0.83	0.05
Import	225.5	5.47	0.34

The import lasted in average 225.5 seconds, about 3min45.

In Step 1, my computer alone consumed on average 0.81 W, i.e. 0.05 Wh during the imports.
In Step 2, my computer with the vagrant containing the application running used 0.83 W. Step 1 and Step 2 are almost equivalent in terms of Wh. In other words, running the application on Vagrant does not consume of lot of energy.
On average, one import used 0.34 Wh. The electric consumption almost did not vary between imports.

If I subtract the noise of the computer, an import alone consumes 0.29Wh. Imports are made 96 times a day, i.e. 35 064 times a year. It means the imports use each year around 10 kWh.

A Danish association, Tomorrow, created an interactive map to see in real time the countries' electricity production and its impact in terms of CO₂. Based on this website and on an 24-hours average, in France, 10 kWh corresponds to 3.5 kg of CO₂ emitted, as our energy comes mostly from nuclear. In the United States, 10 kWh corresponds to 11 kg CO₂.

If the server running my application was behaving exactly like my computer, it would use around 7 kWh a year.

To sum up, I estimated that the application was using 17 kWh each year, and 60% of this energy consumption is made by the imports. The rest is the energy consumes by the server, in the hypothesis of a physical server.

What can we usually do with 17 kWh? With 17 kWh, you can use your air-conditioning for 102 hours, cook 17 chickens in your oven or travel 8.5 km with a small electric car.

It is quite a lot for the server of a small web application.

What can be done to reduce the impact of this application?

I spoke with the developer and the person which was updating the CSV file, and we decided to:

take a quick action and rapidly stop importing the file outside work hours. Importing the file only between 9a.m. and 7p.m. on weekdays reduces the total consumption of the imports to 3 kWh a year, in other words a reduction of 41% of the electric consumption!
in the long term, create a button in the application, which can only be seen by the person updating the CSV file. He would use it when he updates the file, a few times a day. It will reduce the emissions of 95%.

What about the cloud?

Thanks to PowerAPI, I was able to estimate the impact of a costly feature on a physical server. I provided figures to help the project team make decisions to reduce the carbon footprint of the application.

However, this server runs on AWS, and PowerAPI cannot be used in a virtual environment in the cloud. In order to estimate the real impact, we need to find a methodology, using data from the cloud provider: for instance the amount of CPU used by the application.

I will try to see the real impact in the cloud of this feature in an upcoming article!

Flask Debugging in VS Code with Hot-Reload 🔥

Mon, 11 May 2020 00:00:00 GMT

I love using a debugger when I code. It allows me to quickly understand why something does not work as intended, but also to get a faster and deeper understanding of code I did not write.

Since I am so fond of using a debugger, when I started working on a Dockerized Flask application, my first online search was to find how to set one up for my application. But all the solutions I found had different flaws:

❌ The application port would be changed every time I started the application with the debugger
❌ Flask's wonderful hot-reload feature (the server restarting after saving changes to the code) was not supported
❌ The usage was clunky. To make the debugger work, I had to consistently add then remove multiple lines of work

So, I decided to craft my own debugger setup to fix all those problems.

Let's see how to setup a Dockerized Flask app with an efficient debugging flow 🎉!

🔧 Prerequisites

To follow this tutorial, you will only need the following installed:

VS Code
Docker
Docker Compose
The VS Code Python extension

Step 1: Docker setup

To follow this tutorial, make sure you have a Docker configuration similar to this:

# docker-compose.yml
version: "3.4"

services:
  flask-server:
    image: flask
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - 5000:5000
    volumes:
      - .:/app:cached

# Dockerfile
FROM python:3.8

EXPOSE 5000

# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE 1

# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED 1

# Install pip requirements
ADD requirements.txt .
RUN python -m pip install -r requirements.txt

WORKDIR /app

# Switch to a non-root user, please refer to https://aka.ms/vscode-docker-python-user-rights
RUN useradd appuser && chown -R appuser /app
USER appuser

Here we have setup a simple flask-server service that will run our Flask application inside a Docker container.

Step 2: Setup the debugger

VS Code configuration

The only configuration you will need is adding or modifying the .vscode/launch.json file:

// .vscode/launch.json
{
  "configurations": [
    {
      "name": "Python: Remote Attach",
      "type": "python",
      "request": "attach",
      "port": 10001,
      "host": "localhost",
      "pathMappings": [
        {
          "localRoot": "${workspaceFolder}",
          "remoteRoot": "/app"
        }
      ]
    }
  ]
}

Be sure to properly setup the pathMappings property. This will be used to link the files being executed in the Docker container to the files in your machine.
The .vscode folder stores all the project's VS Code configuration files.

Install the debugpy Python module

debugpy is a Python module that will allow you to spawn a debugger inside our Python code.

To install it, make sure to add debugpy to your requirements.txt:

flask==1.1.2
+debugpy

Not specifying the version (ie: not adding a line like this debugpy==1.0.0) will automatically install the latest version of the module.

Use debugpy to create a debug adapter instance

Create a debugger.py file in your application:

# debugger.py
from os import getenv

def initialize_flask_server_debugger_if_needed():
    if getenv("DEBUGGER") == "True":
        import multiprocessing

        if multiprocessing.current_process().pid > 1:
            import debugpy

            debugpy.listen(("0.0.0.0", 10001))
            print("⏳ VS Code debugger can now be attached, press F5 in VS Code ⏳", flush=True)
            debugpy.wait_for_client()
            print("🎉 VS Code debugger attached, enjoy debugging 🎉", flush=True)

Let's explain what is happening here:

Debug adapter logic

debugpy.listen(("0.0.0.0", 10001))

This will start the debug adapter that will listen for a client connection at the 0.0.0.0:10001 interface.

debugpy.wait_for_client()

This line will block program execution until a client (in our case, the client will be the VS Code debugger) is attached.

More logic for a better experience

if getenv("DEBUGGER") == "True":

We want the debugger to be spawned only if the DEBUGGER env variable is set to True. That way, we will still be able to run our application without it.

if multiprocessing.current_process().pid > 1:

In debug mode Flask uses a first process (with pid==1) to start child processes that handle connections. If the code below this line is executed by the main process, the debugging port is taken and subsequent child processes can't use the same port and are attributed a random port which prevents connections.

Instantiate the debugger

Given that your Flask application instance app is created in a file named app.py, call the initialize_flask_server_debugger_if_needed function inside it like this:

# app.py
from flask import Flask
from debugger import initialize_flask_server_debugger_if_needed

initialize_flask_server_debugger_if_needed()

app = Flask(__name__)

@app.route("/")
def home():
    return "Hello, Flask!"

Expose the debugging port

To be able to listen to the debug port of your Docker container, add the 10001 port to the list of exposed ports:

# docker-compose.yml
...
services:
  flask-server:
    ...
    ports:
      - 5000:5000
      - 10001:10001
    ...

Launch your application with a debugger 🎉

Everything is set up, now we only need to start our application!

Here are the docker-compose commands in a Makefile to launch your application with hot-reload using the flask executable.

# Makefile
## 🌶 flask and hot-reload
flask:
	docker-compose run --rm -e FLASK_APP=app.py -e FLASK_ENV=development --service-ports flask-server flask run --host 0.0.0.0

flaskdebug:
	docker-compose run --rm -e DEBUGGER=True -e FLASK_APP=app.py -e FLASK_ENV=development --service-ports flask-server flask run --host 0.0.0.0

The only difference between the two commands is the -e DEBUGGER=True parameter. This will set the DEBUGGER env variable to True inside your container, allowing the debugger code to be executed.

Here's a GIF to show the debugger in action:

To summarize, here are the steps to follow:

In a terminal, run make flaskdebug
When prompted ⏳ VS Code debugger can now be attached, press F5 in VS Code ⏳, press F5
Add a breakpoint to the line you want to debug by clicking left of its number
Call the corresponding route (localhost:5000/hello/flask in the GIF)
The debugger stops the code at your breakpoint, happy debugging 🎉

Hot-reload in action 🔥

Now that it's easy to use the debugger, let's see how the application hot-reload feature is supported.

As you can see in the GIF, after I saved modifications to a file:

The application restarts
The debugger is detached
I am prompted ⏳ VS Code debugger can now be attached, press F5 in VS Code ⏳ again
After I pressed F5, the application is now running with the new changes and the debugger is attached

Tips & tricks

Use the Debug Console to code on the fly

Whenever you are stopped at a breakpoint in your code, VS Code's Debug Console acts as a Python interactive console with the full current context of your code.

Here you can see that I was able to do the following:

Import date from the datetime module
Re-assign the name value to date(3000, 1, 1)
The route response is correctly updated with the modifications

The Debug Console is a very useful tool to quickly test edge cases and find fixes!

Running the app with gunicorn

To use gunicorn instead of flask to run the app, use these commands:

## 🦄 gunicorn and hot-reload
gunicorn:
	docker-compose run --rm --service-ports flask-server gunicorn --reload --bind 0.0.0.0:5000 app:app

gunicorndebug:
	docker-compose run --rm -e DEBUGGER=True --service-ports flask-server gunicorn --reload --bind 0.0.0.0:5000 --timeout 3600 app:app

Running the app as a top-level script

If you need to run your app as a top-level script, ie:

The app is run with python app.py

# app.py
...
if __name__ == "__main__":
  app.run("0.0.0.0", debug=True)

Be sure to make this modification:

if __name__ == "__main__":
    from debugger import initialize_flask_server_debugger_if_needed

    initialize_flask_server_debugger_if_needed()

    app.run("0.0.0.0", debug=True)

Thanks to this, if you want to use the app instance in another file:

# another-python-file.py
from app import app

The code initiating the debugger will not be executed when the import is made.

Wrap-up 🌯

I showed you how to resolve the three problems presented I encountered trying to setup a debugging flow in a Dockerized Flask application:

✔️ The application port does not change
✔️ Flask's wonderful hot-reload feature is supported
✔️ The debugger is easy to use

I created a repository containing a minimal application with everything shown in the article.

I hope that this article helped you setup a (better!) debugging flow of your Dockerized Flask application 💪.

You have a question? No problem, feel free to either contact me:

📧 by email
🐦 on twitter
🐙 by leaving an issue or a pull request on the tutorial repository
💬 by leaving a comment below

I will be very happy to discuss this tutorial with you 😃

Stop losing data when writing Django migrations !

Thu, 07 May 2020 00:00:00 GMT

Saying that database structure is important is sort of an obvious statement. That is of course if you decided to use a structured database technology. But in that case, you want your database structure to be the closest to your functional and business needs, and the tightest in order to be able to rely on that structure.

However, when I was working on Django projects, I sometimes found it painful to change the data structure when I needed to. However, I discovered a very useful tool offered by Django migrations : the RunPython command.

Simple migration

Let’s take a simple example and assume that we have a single model called Order inside an orders application.

We would then have the following models.py file:

from django.db import models


class Order(models.Model):
    reference = models.CharField(max_length=8)
    amount = models.DecimalField(max_digits=8, decimal_places=2)
    creation_date = models.DateField(auto_now=False, auto_now_add=False)
    due_date = models.DateField(auto_now=False, auto_now_add=False)
    customer_name = models.CharField(max_length=50)
    customer_address = models.CharField(max_length=50)
    customer_city = models.CharField(max_length=50)
    customer_zip_code = models.CharField(max_length=50)

    def __str__(self):
        self.reference

Running python manage.py makemigrations will produce the following migrations file, called 0001_initial.py :

from django.db import migrations, models

class Migration(migrations.Migration):

    initial = True

    dependencies = []

    operations = [
        migrations.CreateModel(
            name="Order",
            fields=[
                (
                    "id",
                    models.AutoField(
                        auto_created=True,
                        primary_key=True,
                        serialize=False,
                        verbose_name="ID",
                    ),
                ),
                ("reference", models.CharField(max_length=8)),
                ("amount", models.DecimalField(decimal_places=2, max_digits=8)),
                ("creation_date", models.DateField()),
                ("due_date", models.DateField()),
                ("customer_name", models.CharField(max_length=50)),
                ("customer_address", models.CharField(max_length=50)),
                ("customer_city", models.CharField(max_length=50)),
                ("customer_zip_code", models.CharField(max_length=50)),
            ],
        ),
    ]

Let's break down what's happening in this migrations file. The migration operation is represented as a python class, with 3 attributes:

initial states that this migration is the first of its Django app;
dependencies list all the migrations that need to be applied before this one can be applied. Here it is empty since the migration is the first of its Django app;
operations is the most important part. It is a list of actions that Django will apply to the database. Here the database was empty, so we created the model from scratch with the CreateModel method.

The python manage.py makemigrations performs well when the operations to perform are simple enough. Renaming a model, or renaming a field will be treated as equivalent database operations for example.

After applying the migration, let's create a few entities in the database.

Complex structure migration

Let's start again from our previous example and observe something that you may already have noticed. The 4 fields customer_name, customer_address, customer_city, and customer_zip_code are functionally attached to the Order entity, while they represent another physical entity, a customer. And that is fine as long as there is no duplication. But now let's imagine that for business purposes, you need to abstract a Customer entity to which the Order entities are linked.

Your models.py file would then be:

from django.db import models

class Customer(models.Model):
    name = models.CharField(max_length=50)
    address = models.CharField(max_length=50)
    city = models.CharField(max_length=50)
    zip_code = models.CharField(max_length=50)

    def __str__(self):
        return self.name

class Order(models.Model):
    reference = models.CharField(max_length=8)
    amount = models.DecimalField(max_digits=8, decimal_places=2)
    creation_date = models.DateField(auto_now=False, auto_now_add=False)
    due_date = models.DateField(auto_now=False, auto_now_add=False)
    customer = models.ForeignKey(Customer, on_delete=models.PROTECT)

    def __str__(self):
        return self.reference

This configuration now fits our functional needs. Let's try and generate the migration !

> You are trying to add a non-nullable field 'customer' to order without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
 2) Quit, and let me add a default in models.py

There is a problem however. Django recognizes that we are trying to create a customer field on the Order model, which can't be None but is not defined. So it wants to add a non-null value to all the existing entries in the database, and asks us if we want to provide it now or want to set it in the Order model.

However, none of these options satisfy us. We don't want our orders to have customers that are either None or a default customer, which would lose all the existing data.

Granted, we could also execute this migration, then manually set the customer attribute of all the Order model, but then this forces you to execute this same action on all of your environments. That would also make the migration quite difficult to rollback.

Fortunately, Django comes with an built-in solution to deal with this limitation.

Generating a custom migration

First of, let's start by generating an empty migration that we will then edit.

> python manage.py makemigrations orders --empty

The generated migration file will look like this

from django.db import migrations

class Migration(migrations.Migration):

    dependencies = [
        ('orders', '0001_initial'),
    ]

    operations = [
    ]

We then need to build our custom migration. It needs to have 6 steps :

Create the new Customer model in the database, with all the necessary fields.
Create a nullable customer foreign key on Order
Set the Order fields to transfer as nullable
Transfer the data from the Order model to the Customer model.
Set the new customer field on Order as non-nullable
Remove the old fields from the Order model.

The Django documentation explains all the database operations needed.

Let's write this migration :

from django.db import migrations, models
import django.db.models.deletion

class Migration(migrations.Migration):

    dependencies = [
        ("orders", "0001_initial"),
    ]

    operations = [
        # step 1: add the new Customer model
        migrations.CreateModel(
            name="Customer",
            fields=[
                (
                    "id",
                    models.AutoField(
                        auto_created=True,
                        primary_key=True,
                        serialize=False,
                        verbose_name="ID",
                    ),
                ),
                ("name", models.CharField(max_length=50)),
                ("address", models.CharField(max_length=50)),
                ("city", models.CharField(max_length=50)),
                ("zip_code", models.CharField(max_length=50)),
            ],
        ),

        # step 2: add the nullable foreign key field `customer` to Order
        migrations.AddField(
            model_name="order",
            name="customer",
            field=models.ForeignKey(
                null=True,
                on_delete=django.db.models.deletion.PROTECT,
                to="orders.Customer",
            ),
        ),

        # step 3: set the order fields as nullable
        migrations.AlterField(
            model_name="order",
            name="customer_address",
            field=models.CharField(null=True, max_length=50),
        ),
        migrations.AlterField(
            model_name="order",
            name="customer_city",
            field=models.CharField(null=True, max_length=50),
        ),
        migrations.AlterField(
            model_name="order",
            name="customer_name",
            field=models.CharField(null=True, max_length=50),
        ),
        migrations.AlterField(
            model_name="order",
            name="customer_zip_code",
            field=models.CharField(null=True, max_length=50),
        ),

        # step 4: transfer data from Order to Customer
        ...

        # step 5: set the `customer` field as non-nullable
        migrations.AlterField(
            model_name="order",
            name="customer",
            field=models.ForeignKey(
                null=False,
                on_delete=django.db.models.deletion.PROTECT,
                to="orders.Customer",
            ),
        ),

        # step 6: remove the old Order fields
        migrations.RemoveField(model_name="order", name="customer_address",),
        migrations.RemoveField(model_name="order", name="customer_city",),
        migrations.RemoveField(model_name="order", name="customer_name",),
        migrations.RemoveField(model_name="order", name="customer_zip_code",),
    ]

In this migration, step 2 and 3 basically loosen the data structure, preparing it for the data transfer. Then step 5 and 6 tighten it again !

Let's now dive in step 4, where the magic happens !

Write the data transfer function

In order to perform this operation, we are going to use the RunPython migration operation. What it basically does is execute a python function using the ORM.

Its syntax is the following :

# step 4: transfer data from Order to Customer
migrations.RunPython(order_to_customer, reverse_code=customer_to_order)

In this step of the migration, we specify two functions :

order_to_customer will be run when running the migration ;
customer_to_order will be run when reversing the migration.

def order_to_customer(apps, schema_editor):
    Order = apps.get_model("orders", "Order")
    Customer = apps.get_model("orders", "Customer")
    for order in Order.objects.all():
        customer, _ = Customer.objects.get_or_create(
            name=order.customer_name,
            address=order.customer_address,
            city=order.customer_city,
            zip_code=order.customer_zip_code,
        )
        order.customer = customer
        order.save()

def customer_to_order(apps, schema_editor):
    Order = apps.get_model("orders", "Order")
    for order in Order.objects.all():
        order.customer_name = order.customer.name
        order.customer_address = order.customer.address
        order.customer_city = order.customer.city
        order.customer_zip_code = order.customer.zip_code
        order.save()

These functions look a lot like what you would write to manually transfer the data from one model to the other. However there is one trick. It isn't possible to import the models normally with from orders.models import Order, Customer. Instead, we need to use a versioned model passed to the migration function by Django.

Running the migration

After running this migration via python manage.py migrate, we can check that the migration has created the Customer instances and linked them to the correct Orders.

Customers

Orders

Et voilà ! This migration is now operational. It can also be easily rolled back with python manage.py migrate orders <previous_migration_id>. The data will then be transferred back to the Order model.

Conclusion

You now have a way to stop worrying about losing data when migrating your database with Django! The Django ORM is a great tool and although it does the job most of the time, understanding what happens under the hood is a great way of making your life easier when dealing with migrations.

Sources

Django documentation on migrations:
- https://docs.djangoproject.com/en/3.0/howto/writing-migrations/
- https://docs.djangoproject.com/en/3.0/ref/migration-operations/
Link to the Github repo with the example https://github.com/fargito/django-runpyton

How to Remain Agile with DynamoDB

Tue, 05 May 2020 00:00:00 GMT

Amazon DynamoDB is built to deliver single-digit millisecond performance at any scale. It is built to store Terabytes of data. It is built to support Amazon's Cyber Monday traffic.

However, this scalability comes with an overhead. No matter what resource you look up, you will be constantly reminded that "You Must Know Your Access Patterns In Advance". Amazon Marketplace had been running for years before they migrated to DynamoDB so they were well equipped to know how they needed to access their data and fully reap the rewards of a NoSQL setup.

In contrast, a new startup with an exciting new product wants to build an MVP quickly. They don't know how their app will grow over time. They want to be agile and react to their users to develop their product accordingly. They can't know all their access patterns in advance.

This no doubt poses an issue, but not an insurmountable one.

Being able to map your access patterns in as much detail as possible will translate to a smooth DynamoDB adventure. However, building a startup is about disrupting the market so you can't be afraid of a little turbulence along to way to reach your goal.

If you use DynamoDB from the start you will never have to worry about the scalability of your product. If your app gets a massive social media spike, you won't have to scramble to prevent your site going down. And reaching that scale is what all startups aspire to so how can we build towards this while limiting the turbulence?

Check out my full article on our Serverless Transformation Blog.

Why You Need a Framework When Doing Serverless

Mon, 27 Apr 2020 00:00:00 GMT

We are proud to have been selected as a Serverless Framework official partner. It rewards our investment in the technology and reflects our strong belief that this framework is currently the best in the market.

Using a Framework When Going Serverless is a Must

Recently, one of our clients’ technical advisors shared with us the challenges that he had encountered when testing serverless a few years ago. As he started to use AWS Lambda functions, he quickly lacked visibility of the infrastructure design and struggled to figure out which version of the code was running in production. Fortunately, these issues are now solved by frameworks for serverless applications.

At Theodo, we began leveraging serverless functions a few years ago for projects requiring asynchronous workloads. To avoid the aforementioned pitfalls, we looked into tools that would allow us to be more efficient in managing our serverless projects. The most popular tool was, and remains today, the Serverless Framework.

The main advantage of the Serverless Framework is to allow developers to write to one single repository containing:

The code of the business logic run by their serverless functions,
All the files that describe their application’s infrastructure, including:
- Database services like DynamoDB,
- Authentication services like Cognito or Auth0,
- Notification services like AWS Simple Notification Service or Twilio,
The scripts they need to easily deploy their application in one command to any environment,
Tools allowing developers to emulate services like DynamoDB in their local development environment. Other things that the Serverless Framework achieves is the deployment of a new stack with the whole infrastructure in a single command, allowing to run automated tests consistently on a ephemeral — hence cheap — production-like environment. This further stresses the productivity gains and cost reductions observed by companies embracing serverless.

Why We Chose to Work With the Serverless Framework

Its Large Community Guarantees its Relevance

The suite of serverless services offered by cloud providers is constantly expanding. Therefore, the value of a serverless application framework lies in the ability of its community to stay current.

The Serverless Framework has the largest community among its competitors, and it provides many plugins that gradually add support for new services as they are released. You can read about a few of our favorite tools for this framework in this article.

Its Independence From Cloud Providers

Other frameworks, like AWS’ own Serverless Application Model (or SAM), are open-sourced and maintained by cloud providers themselves — offering early support for their newest features and services. The purpose of some third-party frameworks, however, is to enable developers to deploy their serverless applications to any cloud provider. As the concerns about vendor lock-in grow, the appeal of using a cloud-agnostic framework becomes stronger. It allows you to move your infrastructure to another cloud provider if necessary.

The development of a reliable and powerful framework is a daunting and ambitious task, but we believe in the purpose of the Serverless Framework and we enjoy using it daily on many of our projects. We contribute to the community, for example with sls-dev-tools, our open-source project that gives developers visibility on their serverless application directly from the command line.

With close to sixteen million downloads and hundreds of thousands of weekly deployments, Serverless Framework has become the benchmark in the serverless ecosystem — and this is only the beginning.

Want to know more about how Serverless can help your company? I'd be happy to help. Please reach out here!

Reduce redux boilerplate with redux-toolkit

Mon, 20 Apr 2020 00:00:00 GMT

You're looking for an easier way to write your reducer, aren't you ?

When writing plain redux, you have to define an action, an actionCreator and a reducer to update the store of your app. The reducer is in most cases a big switch to handle all the actions dispatched by actionCreators in your app. This leads to a lot of boilerplate and make the code more difficult to understand. There are many librairies out there aiming to reduce this boilerplate.

At Theodo, we used typesafe-actions to reduce redux boilerplate. The redux team recently recommended @redux-toolkit as the "official recommended approach for writing Redux logic". Thus, I tested it and wrote this article to show you the advantages of such librairies through a simple example.

The example

The example consist in changing the owner of a car :

There is one car and 2 possible owners ("JOHN" or "JANE").

When you click on one of the owner, the car owner in the redux store is changed to match the owner clicked.

Plain redux (28 lines)

With plain redux as on the code below, there are :

an action (UPDATE_OWNER),
an actionCreator (updateOwner)
a reducer (itemReducer) to handle all the actions (codesandbox).

The first simplification external librairies provide is to get rid of the action and avoid the switch in the reducer function.

Both typesafe-actions and redux/toolkit give you two functions to simplify action and reducer : createAction and createReducer

We do not need to keep the UPDATE_OWNER action type anymore in an variable to handle it in the reducer.

The switch disapears

Code is easier to read and understand

Typesafe action (21 lines ~ -25%)

With createAction and createReducer functions there is no action constant UPDATE_OWNER and no verbose switch in our reducer anymore : the reducer reacts directly to the actionCreator updateOwner (codesandbox)

Type-safe action provides also a clearer way to write the reducer using the chain API (codesandbox)

You can even combine actions easily and/or chain action handlers:

const counterReducer = createReducer(0)
  .handleAction([add, increment], (state, action) =>
    state + (action.type === 'ADD' ? action.payload : 1)
  )
  .handleAction(add, (state=> action) => ...);

The code is clearer and easier to understand

Remarks :
- @reduxjs/toolkit provides also a `createAction` function but used in a slightly different manner see here.
- @reduxjs/toolkit `createReducer` function is used as typesafe-action

@redux/toolkit (18 lines ~ -35%)

@reduxjs/toolkit goes further and provides a function createSlice, allowing actions and reducers to be created in the same function. It's very powerfull, give it a try codesandbox.

Moreover Redux toolkit is shipped with ImmerJS, "a tiny package that allows you to work with immutable state in a more convenient way".

No need to duplicate the state with spread operators everywhere: You can directly update the state and ImmerJs will take into account the changes and create a new state from the precedent state + the changes you made. Have a look again to the reducer function 👇🏽

const { actions, reducer: itemReducer } = createSlice({
  name: "items",
  initialState: initialItemState,
  reducers: {
    updateOwner: (state: ItemState, { payload: { user, item } }: UpdateOwnerAction) => {
      // This long code with many ... is replaced by one line thks to ImmerJS
      // return {
      // ...state,
      // [item.id]: { ...state[item.id], owner: user }
      // }
      state[item.id].owner = user;
    }
  }
});

This makes reducer and actions easier to understand. Goodbye "desctructuration ..." 😍.

⚠️ Redux teams recommends to read ImmerJs pitfalls before using it, so read it 😉

In the end of March 2020, the redux toolkit team added 2 api functions (createEntityAdapter & createAsyncThunk) to simplify reducer logic.

In the last example I use the createEntityAdapter function : the entityAdapter "generates a set of prebuilt reducers and selectors for performing CRUD operations on a normalized state structure."

After normalizing my itemState and creating the itemsAdapter, I can simplify my reducer function to use the generic entityAdapter.updateOne to handle the ownerUpdate action (check full code here).

This gives you a very fast way to write your reducer logic.

Have a look at redux-toolkit doc to fully understand the createEntityAdapter API

Conclusion :

Reasons to use typesafe-actions

If you have a lot of actions to handle and one reducer has to react to several actions then typesafe-actions chaining will be right for you

⚠️ Typesafe-actions maintainer is currently updating their API to simpllify the syntax. If you want to contribute. ⚠️

Reasons to use redux/toolkit

Easier to write and understand reducers with ImmerJs
Slicers really concise
The API encourages you to use a normalized state and provides you with practical CRUD function to update data in your store

As redux/toolkit is the recommended library by the redux team, you should give it a try. Slicers + ImmerJS = 🔥 : your code is shorter and easier to understand. Moreover the createEntityAdapter will help you to normalize your state and write a new reducer faster thanks to the many CRUD functions provided.

Demystifying Building Native Modules for React Native

Fri, 17 Apr 2020 00:00:00 GMT

You’ve just released a new native SDK. Could you bring more business to your company by releasing on a cross-platform development framework?

Maybe you’re a developer, longing for a native module of an open-source platform API (or even the third-party SDK above 🙄) to be released. Could you fast-track the development yourself?

You could build your own module. With React Native, this becomes a straightforward process.

Let’s talk about React Native for a second.

Why React Native

For those unfamiliar with React Native: it is a cross-platform development framework allowing you to create native apps for iOS and Android. Using React, you can maintain two platforms using the same codebase, allowing faster development, iteration and knowledge-sharing.

With this framework, we have two sides; JavaScript and Native. Between the two is a bridge, allowing bidirectional and asynchronous communication. This is the power of React Native, on top of a multitude of other benefits.

In 2020, we see the likes of Facebook, Bloomberg and Shopify [1] using React Native to develop their mobile applications, amongst others in the Fortune 500. With over 11 million Javascript developers [2] and more companies switching to React Native, releasing your technology for React Native could bring more growth.

We’ve seen other cross-platform development frameworks like Flutter and Xamarin climb the ranks over the last few years, but React Native is still gaining popularity and ever-increasing its performance as it gets more mature.

So hopefully that's convinced you to consider releasing a Native Modules for React Native if you're releasing Native SDKs. Let’s break down how you’d do that.

Native to React Native

As previously mentioned, communication from the native world to React Native is asynchronous. What this means is that any values must be sent through asynchronous callbacks, promises and events, each sent on a batched message queue. This is the architecture as of April 2020; there is currently a re-architecture of the React Native internals and due to be released mid-2020 [3].

On the native side:

The main thread is responsible for the UI.
The shadow queue is responsible for layout calculations.
Each native module has its own thread (Android shares a thread pool)

On the JavaScript side:

JavaScript VM thread which runs the bundled JS code and sends instructions to the native threads via the bridge.

The two sides communicate over the bridge using the message queue.

JavaScript knows about your native modules at runtime - there’s a JSON representation of each Native Module (consisting of module id, method id and arguments) and we can call methods on the Native Module this way.

For the other direction, Native to JavaScript, we can use promises, callbacks and event to transfer data.

To read more about the React Native internals and the upcoming re-architecture, read this article [4].

Module structure

To get started building your native module, there are a number of tools to help set up the skeleton of your project. For example:

We like Bob, as your library will then come pre-configured with TypeScript and support for Kotlin and Swift. If you've got any other favourites, let us know!

For project structure, if you’ve chosen Bob, the library should now have the following project structure:

- /ios
- /android
- /src
    - index.tsx
- /yourExampleApp
    - package.json
    - ...
- ...

In the iOS and Android folders, you’ll want to have your native SDK code - either available in your project or using git submodules to keep the version control.

Exposing iOS Modules

On the iOS Side, there are two languages at play: Objective-C and Swift. You can write the native module entirely in Objective-C, or write methods in Swift and expose them to Objective-C by using the @objc attribute.

The steps are then as follows:

Create a Bridging Header file (Objective-C, .h file) - Here you'll import:

#import "React/RCTBridgeModule.h"
#import <React/RCTLog.h>
#import <React/RCTEventEmitter.h>
#import <React/RCTConvert.h>

Create a Bridging Command file (Objective-C, .m file), expose modules with RCT_EXPORT_MODULE and Expose methods with RCT_EXPORT_METHOD

#import <React/RCTBridgeModule.h>
#import <React/RCTEventEmitter.h>
#import "MyModule.h"

@interface RCT_EXTERN_MODULE(MyModule, NSObject)

RCT_EXTERN_METHOD(myMethodWithNoParams)

RCT_EXTERN_METHOD(myMethodWithAPromise:(NSString)input
resolver:(RCTPromiseResolveBlock)resolve
rejecter:(RCTPromiseRejectBlock)reject)

+ (BOOL)requiresMainQueueSetup
{
    return YES;
}

@end

Define event emitters, implementing the Delegate pattern.

#import <React/RCTBridgeModule.h>
#import <React/RCTEventEmitter.h>
#import <React/RCTConvert.h>

@interface RCT_EXTERN_MODULE(MyModuleEmitter, RCTEventEmitter)

// Create a singleton for the EventEmitter class

- (id)allocWithZone:(NSZone *)zone {
 static MyModuleEventEmitter *shared = nil;
 static dispatch_once_t onceToken;
 dispatch_once(&onceToken, ^{
 shared = [super allocWithZone:zone];
 });
 return shared;
 }

- (BOOL)requiresMainQueueSetup
 {
 return YES;
 }

@end

For Swift, a good pattern to use is the Observer-Command-Emitter pattern:

This ties together your observer which implements the delegate, the event emitter which triggers events and the commands which will be the methods to expose to JavaScript. In the above code snippets, you've seen the Objective-C .m files for the above.

Exposing Android Modules

The steps for Android are similar:

Create a ReactPackage to declare the modules to expose:

package com.mymodule

import java.util.Arrays

import com.facebook.react.ReactPackage
import com.facebook.react.bridge.NativeModule
import com.facebook.react.bridge.ReactApplicationContext
import com.facebook.react.uimanager.ViewManager

class MyPackage : ReactPackage {
override fun createNativeModules(reactContext: ReactApplicationContext): List<NativeModule> {
return Arrays.asList<NativeModule>(MyModule(reactContext))
}

   override fun createViewManagers(reactContext: ReactApplicationContext): List<ViewManager<*, *>> {
       return emptyList<ViewManager<*, *>>()
   }

}

Create your module extending ReactContextBaseJavaModule to declare the methods to expose using @ReactMethod . Implement listeners to emit events.

package com.mypackage

import com.facebook.react.bridge._
import com.facebook.react.modules.core.DeviceEventManagerModule.RCTDeviceEventEmitter
import java.util.\*

enum class MyModuleError(val errorCode: String, val message: String) {
 INPUT_NOT_RECOGNISED( "INPUT_NOT_RECOGNISED", "Input was not of the correct format."),
}

class MyModule(reactContext: ReactApplicationContext) : ReactContextBaseJavaModule(reactContext), MyListener {
 companion object {
   const val MODULE_NAME = "MyModule"
 }

 override fun getName(): String {
   return MODULE_NAME
 }

 private fun createErrorHandler(promise: Promise): (TokenError) -> Unit {
   return fun(error: TokenError) {
     promise.reject(error.errorCode, error.message)
   }
 }

 @ReactMethod
 fun myMethodWithNoParams() {
    doSomething();
 }

 @ReactMethod
 fun myMethodWithAPromise(input: String, promise: Promise) {
   val handleError = createErrorHandler(promise)

   doSomethingWithInputSucceeds(input)?. let {
     promise.resolve("MY_METHOD_SUCCESS")
   } ?: kotlin.run {
     handleError(ModuleError.INPUT_NOT_RECOGNISED)
   }
 }

 override fun onEventSuccess(result: String, message: String) {
   val body: WritableMap = Arguments.makeNativeMap(mapOf(
     "result" to result,
     "message" to message
   ))
   this.reactApplicationContext.getJSModule(RCTDeviceEventEmitter::class.java)
     .emit("MyModule/eventSucceeded", body)
 }

 override fun onEventFailure(error: String, message: String) {
   val body: WritableMap = Arguments.makeNativeMap(mapOf(
     "error" to error,
     "message" to message
   ))
   this.reactApplicationContext.getJSModule(RCTDeviceEventEmitter::class.java)
     .emit("MyModule/eventFailed", body)
 }
}

Your new module

Now you’ve exposed your native modules and methods to React Native and can continue to build on top of them. What’s left is to import them from NativeModules from react-native and you’re ready to create your app:

In /src of your Native Module:

import {
  NativeModules,
  DeviceEventEmitter,
  NativeEventEmitter,
} from "react-native";

export const { MyModule, MyModuleEventEmitter } = NativeModules;

Now you're ready to go. Just yarn add react-native-myModule and import the above MyModule and emitters to get going.

To launch your development experience and quality to the next level:

Add a React Native deployment pipeline - https://blog.theodo.com/2019/04/react-native-deployment-pipeline/
Add end-to-end tests with Detox - https://github.com/wix/Detox
Add support for offline - https://dev.to/reactnativeradio/rnr-157-building-great-offline-ready-apps-in-react-native-with-josh-warwick

What a great development set up!

Summary

Once that’s all ready, your module is tested, you’re ready to publish. Very straightforward with npm publish[5].

There you have it. A native module which can now be released to the JavaScript tech community - perhaps bringing in more customers for your business or fast-tracking your next development track.

Hopefully this article gave you an overall idea of where to start building your bridge module. If you have any questions, let us know, we'd be happy to help!

For more resources, the following are some good resources:

Sources + links:

How to convert an imperative library to a declarative React component: the Sketchfab Viewer

Fri, 10 Apr 2020 00:00:00 GMT

I love using React for multiple reasons. I like the writing style, I like the tooling, I like the ecosystem.

But sometimes you run into a library that doesn't have a React adapter. It can either be pretty old, kind of niche, closed source, etc.

In this series of articles, we’ll look into how to convert an existing imperative JS library into a declarative React component.

We’ll use the Sketchfab Viewer as an example. Sketchfab develops an amazing 3d suite that includes a very configurable web viewer for use with their models.

You can see that their doc is pretty extensive, if a bit messy. It includes a lot of examples and it took me a long time to run into a use case that was not already covered. Yet you can also see that they use an imperative style of JS that doesn’t fit well into the React patterns.

api.addEventListener("viewerready", function () {
  console.log("Viewer is ready");
});

We are going to build a React function component on top of that library. It will accept a config object to declaratively change a color on the model. It will also expose a callback to react to a click event on a part of the model. And we'll use hooks because they're fun!

You can find the final code for this part here.

Part 1: Initializing and exposing the api object
Part 2: Working on it!

Initializing and exposing the api object

Throughout this project we'll work with this fancy chair, courtesy of the official Sketchfab examples.

Start the project with create-react-app

First things first, let's generate a basic CRA project:

npx create-react-app sketchfab-react

In src/, add a Viewer.jsx file and create an empty component inside.

import React from "react";

export const Viewer = () => <></>;

Add that component inside App.jsx

import React from "react";
import "./App.css";
import { Viewer } from "./Viewer";

function App() {
  return (
    <div className="App">
      <Viewer />
    </div>
  );
}

export default App;

Now let's import the library. Sketchfab provides a package install through npm but that's not always the case for other libraries, so we'll include the source in the head of public/index.html.

<script
  type="text/javascript"
  src="https://static.sketchfab.com/api/sketchfab-viewer-1.7.1.js"
></script>

This import will make window.Sketchfab available globally. If you're using Typescript, you will need to ignore it.

Render the Sketchfab Viewer in an iframe

According to the docs, we need to create an iframe and give it to the Sketchfab constructor to render the Viewer.

The way to do this in React is, in Viewer.jsx:

import React, { useEffect, useRef } from "react";

// Our wonderful chair model
const MODEL_UID = "c632823b6c204797bd9b95dbd9f53a06";

export const Viewer = () => {
  // This ref will contain the actual iframe object
  const viewerIframeRef = useRef(null);

  const ViewerIframe = (
    <iframe
      // We feed the ref to the iframe component to get the underlying DOM object
      ref={viewerIframeRef}
      title="sketchfab-viewer"
      style={{ height: 400, width: 600 }}
    />
  );

  useEffect(
    () => {
      // Initialize the viewer
      let client = new window.Sketchfab(viewerIframeRef.current);
      client.init(MODEL_UID, {
        success: () => {},
        error: () => {
          console.log("Viewer error");
        },
      });
    },
    // We only want to initialize the viewer on first load, so we don't add any dependencies to useEffect
    []
  );

  return ViewerIframe;
};

And you should have the viewer displaying!

Expose the api object to access it imperatively

One of the great things about the Viewer API is that it allows us to transform our model from the browser. Here, we'll try to make the chair red.

First we need to extract the api object and store it. We could put it in the state of our Viewer component, but we'll want to use it higher up in the tree, so we will pass a ref as prop and give it the api.

export const Viewer = ({ apiRef }) => {
	// ...
        success: api => {
          // Store the initialized api inside the provided ref
          apiRef.current = api;
        },
	// ...
};

Now, let's get it inside App.

function App() {
  const apiRef = useRef(null);

  return (
    <div className="App">
      <Viewer apiRef={apiRef} />
    </div>
  );
}

To check that it works, let's try to change the color of the background. There's a method for that!

Let's add this code to App.jsx:

function App() {
  // ...
  const changeBackgroundColor = () => {
    apiRef.current.setBackground({
      color: [Math.random(), Math.random(), Math.random(), 1],
    });
  };

  return (
    <div className="App">
      <button onClick={changeBackgroundColor}>Change background color</button>
      <Viewer apiRef={apiRef} />
    </div>
  );
}

Save, click on the button, voilà!

If it doesn't work, check that you have clicked on the play button of the viewer beforehand to load the model ;)

Change the chair color

Alright, that function is a bit bigger but you'll have to trust me.

function App() {
  // ...

  const changeChairColor = () => {
    apiRef.current.getMaterialList((err, materials) => {
      const plasticMaterial = materials.find(
        (material) => material.name === "Blue plastic"
      );
      plasticMaterial.channels.AlbedoPBR.color = [
        Math.random(),
        Math.random(),
        Math.random(),
      ];
      apiRef.current.setMaterial(plasticMaterial, () => {
        console.log("Updated chair color");
      });
    });
  };

  return (
    <div className="App">
      <button onClick={changeBackgroundColor}>Change background color</button>
      <button onClick={changeChairColor}>Change chair color</button>
      <Viewer apiRef={apiRef} />
    </div>
  );
}

A new color for our chair!

Hooks refactor

To abstract the initialization code and clean up our component, we'll write a custom hook that leverages all the code previously written plus a useState for the api object.

Custom hooks are one of the most powerful tools to keep our code well organized. Once you start using them, there's no going back.

import React, { useEffect, useRef, useState } from "react";

// Our wonderful chair model
const MODEL_UID = "c632823b6c204797bd9b95dbd9f53a06";

const useSketchfabViewer = () => {
  // This ref will contain the actual iframe object
  const viewerIframeRef = useRef(null);
  const [api, setApi] = useState();

  const ViewerIframe = (
    <iframe
      // We feed the ref to the iframe component to get the underlying DOM object
      ref={viewerIframeRef}
      title="sketchfab-viewer"
      style={{ height: 400, width: 600 }}
    />
  );

  useEffect(
    () => {
      // Initialize the viewer
      let client = new window.Sketchfab(viewerIframeRef.current);
      client.init(MODEL_UID, {
        success: setApi,
        error: () => {
          console.log("Viewer error");
        },
      });
    },
    // We only want to initialize the viewer on first load
    // so we don't add any dependencies to useEffect
    []
  );

  return [ViewerIframe, api];
};

export const Viewer = ({ apiRef }) => {
  const [ViewerIframe, api] = useSketchfabViewer();

  apiRef.current = api;

  return ViewerIframe;
};

What we learned

We can use refs to feed DOM elements to libraries that need them.
We can use refs to give back objects to parents. Another solution could have been to use a React Context.
We can use custom hooks to make our code cleaner and more readable.

You can find the final source code here.

Shout-out to Sketchfab for their amazing product and in particular to Arthur Jamain for the great collaboration we had over the last few months.

Can I Use Web Components

Fri, 10 Apr 2020 00:00:00 GMT

The goal of this article is to show you what is a web component, why you can now use it in production, and when you should do so.

For all those who already have some knowledge in web components, I would like to specify that I will be talking about the web component V1 standards as the v0 draft APIs are now removed from Chrome since Chrome 80.

If you’re new to web components, don't worry you can catch up starting from the v1 standards, looking at the v0 specifications and standards might not be useful unless you want to understand the journey of the web components to reach its V1.

But first, you might want to be able to understand the basic concepts of a web component.

Definition of a web component

So first and foremost, what is a web component?

"Web Components is a suite of different technologies allowing you to create reusable custom elements — with their functionality encapsulated away from the rest of your code — and utilize them in your web apps." - MDN

Simply put, the web component is not just a thing by itself. For a component to be called so, different technologies have to be combined.

What are those "different technologies" then?

I will go through the different technologies that make a web component. This article won't be able to encompass all the possibilities of a web component and all its possible implementations. For those who would like to go further into the explanations, I will provide some resources as you read through.

Custom elements

A web component is nothing more than an enhanced custom element. A custom element is a javascript class expanding the HTMLElement class that allows you to create a new HTML tag for your component.

First, you have to create a class as shown in the script below.

class MyComponent extends HTMLElement {
  constructor() {
      super();
      ...
    }
}

We then define a custom element with the previously created class.

customElements.define("my-component", MyComponent);

Your custom element is ready to be used as a tag in any web project.

<my-component></my-component>

let's try it in a simple html document:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>my first custom element</title>
  </head>
  <body>
    <my-component></my-component>
    <script>
      class MyComponent extends HTMLElement {
        constructor() {
          super();
        }
      }
      customElements.define("my-component", MyComponent);
    </script>
  </body>
</html>

And that's it! you have created a custom element. Now if you inspect your HTML document on a web browser you should be able to see something like this:

The main idea of custom elements is that you can create, extend, and reuse them everywhere in your code.

If you are looking for a more in-depth documentation, check the official documentation. I would also recommend this article on customElements that is giving you a good overview of the matter.

Shadow DOM

The next technology that we will be discussing is the Shadow DOM. When you attach a shadow DOM to your custom element you then scope all the CSS, and the HTML in this DOM. The big idea of the shadow DOM is to be able to create templates for your web components that won't be affected by any style or js selectors and therefore keep your code and feature as intended.

Let's try it in our previously created component :

First create a shadow root:

const shadowRoot = this.attachShadow({ mode: "open" });

If you want to know more about the attachShadow mode argument I strongly recommend this article from Leon Revill.

Create some elements:

const spanElement = document.createElement("span");
spanElement.textContent = "Hello world";

Create a style element:

const style = document.createElement("style");

Create a css class for your span element and add some css rules:

style.textContent = ".my-class {color: red}";
spanElement.setAttribute("class", "my-class");

And finally attach them to your shadow root:

shadowRoot.appendChild(style);
shadowRoot.appendChild(spanElement);

so here it what your component should look like now:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>my first custom element</title>
  </head>
  <body>
    <my-component></my-component>
    <script>
      class MyComponent extends HTMLElement {
        constructor() {
          super();
          const shadowRoot = this.attachShadow({ mode: "open" });

          const spanElement = document.createElement("span");
          spanElement.textContent = "Hello world";
          spanElement.setAttribute("class", "my-class");

          const style = document.createElement("style");
          style.textContent = ".my-class {color: red;}";

          shadowRoot.appendChild(style);
          shadowRoot.appendChild(spanElement);
        }
      }
      customElements.define("my-component", MyComponent);
    </script>
  </body>
</html>

Now you can try going to the console and access and modify the elements in the shadow DOM but you won't be able to.

As well if you add some style in your HTML file, it won't affect the shadow DOM elements. Try adding style on the span element for example:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>my first custom element</title>
    <style>
      span {
        color: blue;
      }
    </style>
  </head>
  <body>
    <span>Hey! I am just here for the example tho...</span>
    <my-component></my-component>
    <script>
      {...}
    </script>
  </body>
</html>

As you can see the span inside the shadow DOM is not affected by the style.

HTML Template

"The template element is used to declare fragments of HTML that can be cloned and inserted in the document by script. In a rendering, the template element represents nothing. The template contents of a template element are not children of the element itself." - html.spec.whatwg.org

This means that you can define a template for your Web component and clone it for every instance of your component.

So first you create your template:

const template = document.createElement("template");
template.innerHTML = `
  <style>
    span { color: red; }
  </style>
  <span>Hey There! I am in the template </p>
`;

Then you can clone it's content into your shadow tree:

shadowRoot.appendChild(template.content.cloneNode(true));

As I am sure you noticed, we don't just add the template in the shadowRoot but instead we copy its content. That's because Templates are not rendered so what we need for our component is the content of the template.

Let's implement it in our component:

export class MyComponent extends HTMLElement {
  constructor() {
    super();
    const shadowRoot = this.attachShadow({ mode: "open" });

    const template = document.createElement("template");
    template.innerHTML = `
    <style>
        span { color: red; }
    </style>
    <span>Hey There! I am in the template </span>`;
    shadowRoot.appendChild(template.content.cloneNode(true));
  }
}

The important point to keep in mind about templates is that it allows you to declare chunks of HTML in your document that will be rendered and that sole purpose is to be cloned and reused. It is a perfect fit for web components.

You could define it in your index.html since template elements are not rendered in the browser but I would recommend keeping it close to your component declaration to keep your index.html file clean and organize your templates by component.

You can even use the slot Element to make your template more flexible. Those of you who already worked with Vue.js may be familiar with the concept of slot. For the others Slots are like placeholder. They allow the developper to create specific elements that will be placed inside the parents component DOM at predifined emplacement.

Imagine that your component template is a text with holes, each hole has an id. When you add children to your component, you can assign them a slot attribute. This slot attribute allows your component to place them in the "holes " according to their ids (the slot attributes).

If you want to learn more about slot element with web component I would recommend this documentation from MDN.

The addition of those four standards-based technologies is what defines the web component today.

ES Modules

This part is not always explicit in the definition of a web component. It doesn't affect the component on itself but more the way you will be able to use it in a project.

"The ES Modules specification defines the inclusion and reuse of JS documents in a standards-based, modular, performant way." - webcomponents.org

The ES modules should allow developers to use the script tag with the type module. Wich means you can export variables or components from your .mjs files and import it in your .html file within a module script tag.

The ES modules standard allow developers to reuse web component in any HTML file by importing them from an mjs file.

// wc.mjs
export class MyComponent extends HTMLElement {
  constructor() {
    super();
    const shadowRoot = this.attachShadow({ mode: "open" });

    const spanElement = document.createElement("span");
    spanElement.textContent = "Hello world";
    spanElement.setAttribute("class", "my-class");

    const style = document.createElement("style");
    style.textContent = ".my-class {color: red;}";

    shadowRoot.appendChild(style);
    shadowRoot.appendChild(spanElement);
  }
}

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>my first custom element</title>
    <style>
      span {
        color: blue;
      }
    </style>
  </head>
  <body>
    <span>Hey! i am just here for the example tho...</span>
    <my-component></my-component>
    <script type="module">
      import { MyComponent } from "./wc.mjs";
      customElements.define("my-component", MyComponent);
    </script>
  </body>
</html>

An MJS file is a source code file containing an ES Module.

You will need a server to be able to fetch with import, as it doesn’t work on the file:// protocol. You can use npx serve to start up a server in the current directory for testing locally.

If you are interested in how the ES module works I strongly advise that you read this article from Lin Clark.

Can I use it?

Now, a lot of developers are still quite reluctant to use web components in their projects. And you can understand why! The lack of common standards implemented by all major browsers since it's creation in the early 2010s has allowed many web component libraries to raise and grow sub-communities that do not share the same opinion on the definition of a web component and its implementation.

On the article Why the React community is missing the point about Web Components by Ben Halpern in November 2018, Dan Abramov comments the following :

[...] The problem is that there's no single "web component community". There are multiple sub-communities. You mention "web component libraries" in the post. These libraries don't agree on a single standard so React would need to pick a side in debates like "how does server rendering work with web components". Whatever we pick will have large downstream effects, and our reluctance to support more has to do with being careful [...] about the semantics — not somehow being at odds with web components per se.

So I think some important points have to be clarified for you to understand the implication of using it in your projects.

Web components are tools

It is important to understand that Web components are not designed to be better than HTML elements. We can consider web components as a tool to extend HTML, not to replace it.

And as we know the same tool cannot be used for all projects. So before adding this tool to your toolbox make sure you really need it or else it might some unecessary weight.

Here are some points that you should consider before jumping in :

It is not that simple to write a web component. It can be quite complex and verbose depending on the scope of your web components. And most of the project with quite a large number of web components have been using some libraries to creates them. webcomponents.org has a page dedicated to those libraries.
Comparing a web component and a react component for example is like comparing apples and oranges. The definition of a web component is not setting limits to what you can put in it. React in its documentation explain it this way:

"React and Web Components are built to solve different problems. Web Components provide strong encapsulation for reusable components, while React provides a declarative library that keeps the DOM in sync with your data. The two goals are complementary. As a developer, you are free to use React in your Web Components or to use Web Components in React, or both."

Understanding and defining what will be the interfaces between web components and the rest of your project's tools is important.

Web component v1 standards are not that old and you probably want your project to work on older browsers versions too. Simply put, you might need polyfills. I will come back to it later on.
One of the down points of the web component often cited is that developers have been struggling to use it with server-side rendering. This is mainly caused by the fact that the contents of ShadowDOM are often missing when the view is delivered to the client. One possibility is to remove the shadow dom from your component like you can see in this article. Other web component advocates found ways to make it work like in this article by Steve Belovarich or this one by Peter Goes. I think you can sum it up by saying that for now there are no standards for server-side rendering with a web component but you can always find a workaround.

Just remember to take all this point in account before jumping in.

Future proof

One of the main arguments of the pro web component community is that web components are future proof. Since it is based on web Standards, it is now supported by all modern browsers and should always be.

I can already hear your ask:

"Alright that's for the future? But what about our beloved past heroes like IE?"

Well you are right to be worried about them and that should be one of your main concerns.

Below are the previously cited technologies browser supports at the current date. If you want the latest data, click on the link below each of them.

can i use custom element v1

can i use custom shadow DOM v1

can i use javascript module script tag

can i use javascript module dynamic import

can i use HTML templates

As you can see neither Internet explorer is not supporting any of them neither the early versions of other browsers.

There are polyfills for the web component APIs. You can use the official polyfills.

With those polyfills, nothing you can now develop your future and past proof web component!

Reusability

A web component is easily reusable and is trustworthy. Its strong encapsulation provides security for its data. It also ensures that it will be used the way its designer intended to.

Should I use it?

Based on previous points, there are no specific reasons to not go for it. Still, the Web component isn't mature enough to be trusted for production by most developers. It solves problems but adds others.

For now, the best use the community can agree on for a WC-based project is a design system UI components library. Tesla for example, has crossed the bridge and chosen to make its design system with web components.

Building a sustainable design system can be a real brain teaser for companies. Big companies often use a lot of different technologies and frameworks for all their products.

They reach a point where sustainability becomes a problem in terms of design and technology. They ask themselves those questions :

Do we have to create a UI component library for each framework that we use?
How can we be sure that we don’t switch framework in the next few years?
And If we make sure we don't, aren't we missing an opportunity?

A web component is not based on any framework owned by any company but on standards adopted by all major browsers. You don't have to worry about the hype effect on your chosen technology or framework. This is why web components fit if you decide to create a UI component Library or even a design system.

Of course, you can also use it in any static pages project but I wanted to highlight the one big use where you should consider it.

Conclusion

Now that the standards that define Web components are implemented in every modern browser, they are meant to be part of the future of the web.

As a web developer, it’s time to consider web components as a practical solution for your features and problems.

I would like to quote Serhii Kulykov(@webpadawan) on this article

"Now in 2019, it’s time to give a try to Custom Elements and Shadow DOM. They might not be “the next big thing” by themselves, but rather solid foundation the next big thing could be built upon." - Serhii Kulykov

Of course when the technology will be more mature, accepted and integrated into our projects we will witness the apparition of new ideas and maybe even new standards.

But for now, we are the ones who have to be bold enough to try on our own.

If you are interested in the subject, I recommend the following articles:

The journey of Web Components: wrong ways, lacking parts and promising paths - Serhii Kulykov

Why I don't use web components? - Rich Harris (svelte creator).

I recommend you read the article but most importantly to read its comment. Especially this gist that is a direct answer to the previously mentioned article

Why the React community is missing the point about Web Components - Ben Halpern.

Once again the interesting part is mostly in the comments with a discussion between Dan Abramov et Benny Powers