Theodo

Write tests for humans

Mon, 07 Sep 2020 00:00:00 GMT

Let me tell you the story of the day I almost quit testing my code altogether. For the tenth times this week, the CI failed with this cryptic error message "Received value does not match stored snapshot" and a diff that looked something like:

<div
  - className="myClassName" >
  + className="myNewClassName" >
  + <div>
    Hello world
  + </div>
</div>

It felt pointless to fix this test : why does it matter if the class-name is different or I added a div tag? It does not keep me from introducing regressions as nobody in the team reads the snapshots updates.

I was so frustrated, I almost broke the computer, slammed the door to quit my job. But instead, I decided to channel this energy into something more positive. I started thinking about how the project tests got into such a state.

How come everybody agrees that we need to test our code base yet so many developers hate it? Was I the only one being frustrated? (No I was not)

When people say they use TDD, I don’t believe them.
— emily freeman (@editingemily) August 26, 2020

Why is nobody doing something about this?

The Vicious Circle Theory or How you stopped writing test for humans

Here is my theory to make sense of this kind of situation. I have called it "The vicious circle of testing" ™:

A Developer changes the code for a feature. A test fails, but it is not clear why.
The Developer fixes the test. They do not know why the test was here in the first place. They grow used to the idea that it is not worth their time investment to fix it.
Next time, this developer writes code, they spend less time and effort writing tests.
And back to the top...

If this resonates with you, you've been writing tests for the CI to work but not for your teammates to understand your intent. Tests are a collaboration tool, they are meant to be read when they fail by you of your teammates. You need to make sure you can understand them in a month when you have forgotten all about what you were supposed to do.

Tests are meant to be read by humans, not your CI.

How do you write tests for humans, you ask? Here are my 3 tips for you :

Why: You need to know why you are testing. Because testing is a universal truth in software engineering, it is easy to forget why you need to do it on your project. Find what motivates you: it will guide you to write good tests that you want other humans to read.
What: You need to focus. I think it is better to do one thing well rather than try to do it all and do it mediocrely. Choose what you want to test, do not try to test it all.
How: Choose the best tools adapted to your goal. They must help you write quality tests that you can read and maintain

#1: Why do you write tests?

Writing great tests for humans takes time, you need to be invested in what you are doing. You cannot be if you do not know exactly why you need to write great tests.

Do not write tests because you have to. Make sure you and your team know why you write tests.

Automatic testing is often a given when setting up quality tools for your project. Unlike your linter, tests do not fail every day, they are a longer-term investment to code quality. It is easy to lose sight of why you are doing it.

Before starting to set up your test environment you should clarify why you are testing.

What are you hoping to gain in the long term? Better documentation? Fewer regressions bugs? Developer experience tool for TDD?

Make sure you and your team know why you are doing it. Because it is so easy to fall into the vicious circle of bad testing, if you all know why you are driven to write tests, you can catch early when you are deviating from your goal. Making your goal clear will give your teammates ownership over your test and they will help you push for improvements when needed.

#2: Choose what parts need to be tested first

If we chose to use snapshots on my project, it was to go faster. We wanted to test all of the codebase without investing much time. By aiming to keep a 100% coverage, we started writing tests that needed to be maintained every time we added features. They could not help us catch regressions because they changed too often. No matter how small the investment was, it was not worth it.

Do not aim for a 100% code-coverage of generic tests, aim for 100% of readable tests

You do not need to test everything. Your set up may not be adapted to every aspect of your code.

I always write unit tests for very logical functions :

They are hard to read and easy to break. A test is a safeguard to make sure another developer will not break it.
It is cheap to set up and to write.
It is fast so it does not slow down the ci.

But for most part of the code, it is a case-by-case study.

Should I test my frontend app UX? If I am writing a complex single-page app, UX is very likely to evolve. Testing it may be my priority so I can prevent regressions that have a big impact on the user's appreciation of my app. I want to write integration or end-to-end tests for this.

Should I test your frontend app UI? If I am writing a shared-component library, I am focusing on UI and have a few logical components. My first mission is to provide my users with reliable and consistent UI. Testing UX logic is a bonus, it is ok to make choices and not to focus on this at first.

#3 Choose your tools right

There are a lot of great testing tools available. Now that you know why you want to write tests and what parts of code you want to focus on, you can find the tools that fit your needs like a glove

On another project, I chose to test my frontend React app with react-testing-library. I wanted a tool that :

would test our React app components logic and document what they did
would help new developers feel confident when starting on the project
would help us focus on these goals even if the teams changed

That is why I chose react-testing-library to write integrations tests on it. It turned out to be a great developer experience. I also had a personal goal of trying this new tool that had a lot of great reviews.

Find a tool that makes writing test fun, or that makes you want to do test-driven-development, you will only write tests for humans this way.

You are ready!

Now you know: if you ever find yourself thinking testing is useless or painful to maintain, you are probably in a very vicious circle of testing. If you want to escape this, here are the steps to follow:

Remember tests are a documentation and collaboration tool, they should be made to be maintained by teams and read by humans
Make sure your team knows why you are writing tests. This will motivate you to write good tests that you can be proud of.
Do not try to do it all. Coverage is just a number, it is not a token of code quality. Focus on testing some parts well rather than all of it in a poor way.
Choose the tool that fits you and that makes the developer experience fun

Software-based CPU power consumption using PowerApi

Thu, 03 Sep 2020 00:00:00 GMT

The ICT sectors carbon footprint represents now around 4% of the global CO2 emissions. To minimize that, one of our goals at Theodo is to design sustainable web applications with minimal carbon footprint. And in order to know how performant we are at this, we need to be able to evaluate the power consumption of our applications and more specifically the power consumption of the VMs we use in the cloud. Unfortunately, at the moment no cloud provider is able to provide such a data. So we had to look for a solution.

Our research led us to PowerApi, an open-source project developed by the Spirals research group from University of Lille 1 and Inria in France. It allows people to estimate the CPU’s power consumption induced by an application without any external device like a wattmeter. As we can't put wattmeters to measure the VMs consumption in the cloud, this looks like a good option.

The magic formula

From the documentation, we can read that PowerApi collects with a sensor raw data from the hardware of the server. Then PowerApi applies a “formula” and voilà: we have the power consumption of the monitored software stored in a database.

After reading this I asked myself two questions : what is this “raw data” and how did they build this magic formula ? Fortunately, we can all have access to their research papers where they explain everything. So let’s dig a bit deeper.

A Power Model is used to approximate the power consumption

The formula is what is called a Power Model in the academic field. A power model is a model that approximates the power consumption of a hardware based on different factors.

I knew that each CPU has a specific TDP on their specification page: “Thermal Design Power (TDP) represents the average power, in watts, the processor dissipates when operating at Base Frequency with all cores active under an Intel-defined, high-complexity workload”

So when I pictured the formula, I thought it was going to be quite simple and something like:

Consumption = Idle power consumption + %CPU load * (TDP - Idle power consumption)

Reading their research paper, it turns out it’s more complicated: “the CPU load does not accurately reflect the diversity of the CPU activities. In particular, to faithfully capture the power model of a CPU, the types of tasks that are executed by the CPU have to be clearly identified.”

Power consumption is related to the CPU activity which is not exactly the CPU load

The types of tasks in a CPU are also known as events. Each processor type has hundreds of different events with potentially a different power consumption. To have an acute idea of what the CPU is doing and thus knowing the related power consumption, the PowerApi teams decided to rely on those events and count them.

To do that they rely on specific counters of the CPU: the HPCs (hardware performance counters) : “We therefore decide to base our power models on hardware performance counters (HPCs) to collect raw, yet accurate, metrics reflecting the types of operations that are truly executed by the CPU.”

Learning phase: creating the Power Model from the regression analysis of real power consumption

The goal of this phase is to approximate the amount of power consumption per HPC events during a workload. It’s made possible by measuring both real Power consumption of the server and counting HPC events in real time.

Step 1: select an unbiased set of workloads

To ensure that the data is not biased to one kind of CPU activity, the team used an academic set of different workloads called PARSEC.

“To explore the diversity of activities of a CPU, we consider a set of representative benchmark applications covering the features provided by a CPU. In particular, to promote the reproducibility of our results, we favor freely available and widely used benchmark suites, such as PARSEC.”

Step 2: get the correlation between real power consumption and HPC events

The team measured the real power consumption using a bluetooth power meter : the Power Spy 2.

Real time in the context of this experiment is that every 250 ms:

The Power Spy 2 send the power consumption of the last 250 ms to a collector
The number of HPC events of the last 250 ms are sent to a collector

By doing so we obtain a lot of data linking power consumption and HPC events.

Step 3: select the HPC events that are the most correlated to power consumption

The two final steps are a mathematical ones. Step 3 is about removing the HPC events that are weakly correlated with power consumption to simplify the step 4.

Step 4: Create the Power Model using ridge regression

At this step, the team uses an advanced mathematical method of regression (ridge regression), to get the 2 or 3 HPC events that will be able to predict the power consumption of the software on the machine. This is what we call the Power Model

Here is a diagram of the whole phase:

The final Power Model includes memory, disks, motherboard and fans

As the Power Spy 2 measures the power consumption of the whole server, all other hardware components (memory, disks, motherboard and fans, ..) are integrated into the power model. It works well because the CPU is the most energy consumming component.

The final power model is P = Pidle + Pactive, where Pidle corresponds to the power consumption of the CPU when no program is running and Pactive to the additional power consumption from a workload.

For example, one power model for the Intel Xeon W3520 is

With e1 and e2 two HPC events.

To get a real time power consumption of this server PowerApi has then just to count those events.

Real life application

As Cyrielle has explained in this blog post, we were able to measure the power consumption on our local machine. But we couldn't use it to calculate power consumption on AWS servers which was our original goal. Indeed the research team can't do the learning phase on the AWS servers (yet?) as they don't have access to it. So the PowerAPI software can't get the data needed for its Power Model.

Thanks to this project and the things we read to try it, we learnt a lot:

The CPU is the component in a server that consumes the most (~100 Watts). The memory is around 10 Watts for 32GB and a SSD consummes around 2 Watts.
CPUs, when idle still consume around 1/3 of the peak consumption.
The next generation of Intel's CPU may see a 91% reduction in power consumption
ARM processors have a very low energy consumption compared to x86 processor(~2 Watts vs ~100 Watts)
AWS provides arm-based EC2 instances if you look for efficient way of reducing the carbon impact of your web application.

The next idea that looks promising to have a rough idea of the power consumption of our application is the "Cloud Jewels" approach by Etsy adapted for AWS.

Prevent AWS from Reading Your Step Functions Data

Mon, 31 Aug 2020 00:00:00 GMT

AWS Step Functions is the perfect tool to handle long-running, complex or business-critical workflows such as payment or subscription flows. However, a naive implementation could put sensitive data at risk.

What is AWS Step Functions?

AWS Step Functions is an Amazon cloud service designed to create state machines to orchestrate multiple Lambda functions, branching logic and external services in one place. It presents most of the advantages of the serverless services: the scaling is immediate and the pricing is based only on the computing time and the number of step transitions.

The state machines can be defined directly in the AWS Console with a JSON configuration and you directly have a graph representation of your workflow.

AWS Step Functions state diagram

At Theodo, we use Serverless as our framework with the Step Functions plugin to write our AWS Lambda functions, define our resources (DynamoDB and Queues) and setup AWS Step Functions and the state machines directly in our codebase.

Example: an identity verification workflow with Step Functions

The previous picture shows an identification verification workflow. It is a piece of a bigger validation flow for banking applications.

What it does is request an identification URL to a third-party system, send this link by text message to the applicant and wait for them to complete the identity verification. When it is done, the third-party system calls a webhook, which restarts the state machine and either triggers a retry, sends a rejection email, or continues the global validation process.

The identification flow

To perform all of those actions, sensitive information will flow through the inputs and outputs of the state machine steps:

👤 Personal information from the applicant which are sent to the third-party service to check his/her identity
📱 The phone number of the applicant to send the SMS
✉️ The email address of the applicant to send the mail
✔️ The result of the identity verification

Problem: we are exposing sensitive data

With this implementation, we see that sensitive data flow through the inputs and outputs of your state machine directly.

state-machine.yaml
----------

SendIdentificationRequest:
  Type: Task
  Resource: arn:aws:states:::lambda:invoke
  Parameter:
    FunctionName:
      Fn::ImportValue: ${self:provider.stage}-sendSMS-Function-arn
    Payload:
      creditApplicationId.$: $.creditApplicationId
      creditApplicant.$: $.creditApplicant
      phoneNumber.$: $.phoneNumber
      message.$: $.smsBody
  ResultPath: $.identificationSMS
  Next: WaitIdentification

An exemple of sensitive data flowing through our inputs and outputs

The problem is: nowadays, most DevOps engineer, developers and even some product managers with access to AWS and can see that information. Not very GDPR friendly, right? Furthermore, AWS Lambda dumps are stored in an S3 bucket which is not encrypted with the default configuration. Any malicious access to your AWS stack could leak sensitive information far more easily than by corrupting your database for example. The more you spread sensitive information between multiple services, the more you put yourself at risk. Finally, as it is stated in AWS documentation:

Any data that you enter into Step Functions or other services might get picked up for inclusion in diagnostic logs.

With that in mind, you may completely dump AWS Step Functions, or you may try to find a solution because the tool is still awesome and perfectly suits your needs!

Our solution

To prevent leaking sensitive information, we need to clean every step function input and output. Here are two ways of doing it:

Keep the data in the input and output but encrypt and decrypt it between each step
Never transmit sensitive data between steps and retrieve it directly during their execution

Both solutions are viable. However, we prefered the second one because we wanted to keep our sensitive data in a single safe spot, where we put a lot of attention to the security.

We were already using AWS DynamoDB for the waiting tokens of our lambda functions, so we decided to create a new table to store sensitive data between steps.

We configured it like below, using an external KMS key.

resources.yaml
----------

StepSentiveInputTable:
  Type: AWS::DynamoDB::Table
  DeletetionPolicy: Retain
  Properties:
    TableName: step-sensitive-input-table
    AttributeDefinitions:
      - AttributeName: sensitiveInputId
        AttributeType: S
    KeySchema:
      - AttributeName: sensitiveInputId
        KeyType: HASH
    BillingMode: PAY_PER_REQUEST
    PointInTimeRecoverySpecification:
      PointINTimeRecoveryEnabled: true
    SSESpecification:
      SSEEnabled: true
      SSEType: KMS
      KeyId: hsm-key-id

DynamoDB table definition in Serverless

We were already using DynamoDB from our lambdas to save our waiting task tokens and we chose to add another table to save sensitive data between steps. Furthermore, on AWS, you can choose to save the encryption keys of your DynamoDB tables outside of AWS, which was a big plus for us.

To avoid repeating ourselves, we coded a wrapper that we use on all our lambdas. It does the following things:

After the return of the lambda, the wrapper:

Looks for sensitive data keys in the output, remove them and build an object with that information
Save the sensitive data objects in a dynamo entry
Add the id of the new dynamo entry under a “sensitiveData” key in the output

Before the execution of the main lambda function, the wrapper:

Looks for “sensitiveData” keys in the input
Retrieve the content at the given IDs
Replaces the "sensitiveData" keys with the resulting object in the input

// encryption-wrapper.ts

export default (lambdaHandler: LambdaHandler): LambdaHandler => {
  const decryptHandler = async (event: any) => {
    const decryptedInformation = await getSensitiveData(event.sensitiveInputId);
    return lambdaHandler({ ...event, ...decryptedInformation });
  };

  return async (event: any) => {
    const clearOutput = await decryptHandler(event);
    const sensitiveInputId = await storeSensitiveData(clearOutput);

    for (const sensitiveKey of sensitiveInputKeys) {
      delete clearOutput[sensitiveKey];
    }

    return { sensitiveInputId, ...clearOutput };
  };
};

Our custom encryption wrapper

Last of all, we needed to encrypt the first input at the start of our state machine from the API to finally secure our users information all the way!

Takeways

Take care about what information you have in your lambdas and state machine events on AWS
Always think twice about the data you put in your logs
Spread your secrets and security implementation in as few services as possible
Use a wrapper if you want to repeat behavior in multiple lambda functions.

Thank you very much for reading my article. Feel free to comment, ask for implementation details, and feel free to share the problems that you encountered with AWS Step Functions.

Typescript: use the nullish coalescing operator to prevent bugs

Tue, 04 Aug 2020 00:00:00 GMT

My goal as a CTO is to improve quality. The score of this game is the number of bugs we find each week. Today I share with you a typical bug that more than one person got caught by.

Let's say you want to initialize the audio volume of your react application with the value previously saved in the localStorage or defaulting it to 0.5 if nothing was saved. You could write something like:

Bad example

function initializeAudio() {
  let volume = localStorage.volume || 0.5;

  // ...
}

The problem is that if the user has saved their volume to 0 it will be interpreted as falsy and your code will use the default value instead of the right value. Some devs prefer the simplicity of || than putting an explicit if clause. And more than once, they were right because 0 was not a plausible value. But as a standard it is too dangerous. For example someone else could see the code and think the || is a good coding stantard in all situation which will eventually create a bug.

Typescript 3.7 comes with the best option: you can now use the nullish coalescing operator to prevent that wrong behavior and safely write something like:

Good example

function initializeAudio() {
  let volume = localStorage.volume ?? 0.5;

  // ...
}

That will be compiled in:

function initializeAudio() {
    var _a;
    var volume = (_a = localStorage.volume) !== null && _a !== void 0 ? _a : 0.5;
    // ...
}

To make sure people from your team use it, you can use the prefer-nullish-coalescing ESLint rule. You can also use it with javascript and this babel plugin. If you want to go deeper in the understanding of this operator you can go here.

I hope this article will help your team prevent this kind of bugs. For more actionable code quality tips, find me on Twitter.

Get rid of bugs due to front-back and back-back communication, by implementing contract testing with Pact

Sun, 05 Jul 2020 00:00:00 GMT

Image from the Pact documentation

Exec summary

Modern software development organizations scale their development efforts by spreading the development of a system across different teams. Such projects can have team work issues because interconnected applications that change often, create a lot of errors breaking other teams' application. A typical problem due to a large organization is when the backend team deploys a change that is not compatible with the mobile app and breaks it.

Contract testing is a way to make sure that several applications developed by collaborating teams can understand each other. With no need to take extra manual care when modifying code, the teams avoid human errors, get rid of bugs, and on the way, develop faster.

With contract testing, a team can easily create a test checking that the application they are developing receives viable responses from the provider application that another team is developing. On their side, the provider application team only has to run a single automatically generated test. This allows both teams to know if something is broken before they deploy their applications: they can talk quickly about this precise problem, with little rework cost and no bug for the users.

Pact is the main framework to implement contract tests, and it works with many languages.

For Whom?

In order to avoid creating bugs coming from miscommunication between two parts of your applications you can set up different tactics, contract testing is one of them. Here are some alternatives with an evaluation of how they perform. You can find some comments on these alternatives in the appendix at the end of the article.

Alternative	Maintenance cost	Setup cost	New interaction cost	Learning curve	Efficient
End-to-end testing	$$$	$$$	$$	$$	++
Shared types	$	$$	$	$	++
Versioned API without breaking change	$$$	$$	$	$$	+++
Massive documentation and conception	$$$	$	$$$	$$$	+++
Contract testing	$	$$	$	$$$	+++

All these tactics can be used together where it is possible.

In fact contract testing needs some conception to create the contract or update it. But contract testing allows to have an accurate documentation when you want to do this conception. Once you discover a breaking change you still have to take a decision on how to handle it and versioned API are a good way to do it but the contracts will show you what you do not need to maintain because no one uses it anymore.

Conclusion: if you develop a distributed system communicating via HTTP APIs with heterogeneous technologies you should find someone to help your team set up contract testing.

Benefits

Benefit #1: Defects are stopped before wasting others time

Usually to achieve testing of several services interacting, we use some kind of end-to-end testing. This means you can test only after deploying your service to put it on the same environment as other services (e.g. staging, preprod, CI). What is called end-to-end testing here could be called broad stack tests, it matches the definition found here. If there are several teams, it becomes difficult to have someone responsible for end-to-end tests.

With contract testing you can test directly in your continuous integration (CI) workflow which means no one else than the developer invested time in checking the work done, this allows problems to be caught earlier. (cf. process just below)

doing > manual local test (few minutes) > push (few seconds) > CI (few minutes) > code review (some minutes) > merge (few seconds) > deploy (few minutes) > manual test on staging > validation by the PO

Benefit #2: Developers know their mistakes and correct them quickly

As you can see on the process above having the result in CI instead of after deployment creates a faster feedback loop for the developer and he can correct it before the merge which means corrections are done faster also.

Let me introduce the pyramid of tests briefly, there are several kinds of tests with different drawbacks, the idea is that even if useful, the more drawback you have, the less tests of this kind you do.

Image from the Martin Fowler article Test Pyramid

The main advantage is that contract tests are lower on the pyramid of tests (between unit for the cost and service for the speed) than end-to-end tests (UI). It means they are faster and cheaper, so we can test more cases which gives quickly more confidence for less money.

Benefit #3: Contract tests are easy to create and maintain

With the pyramid we saw that contract testing is cheap compare to the end-to-end alternative. Our experience is that once set up the cost is close to unit tests (only slower) which are widely used on most projects.

Each test, tests exactly one case, this means they do not change often (compared to end-to-end tests where any change can break the test) and that a change is easy to do in the test because the complexity is low (like a unit test).

Benefit #4: Contract tests give auto-documented API usage

Thanks to Pact that registers all your contract tests, you know precisely what are the interactions tested in your project. This is up to date even before going to production and you can easily visualize it from Pact.

Services interactions:

Image from the Pact documentation

Details of an interaction:

Image from the Pact documentation

Trade-offs

Trade-off #1: It is useful only when several teams work together

Tests are used to perform check that are hard to remember and take a lot of time to perform manually. For contract tests this means they are less useful when you only have one or two teams as you usually know exactly what the other team uses.

They remain relevant but the cost for an inexperienced team on how to use them could create more problems than it solves at first. For an experienced team the cost is not an issue making them a practical solution even to test the interactions between one frontend and its API developed by the same team.

Trade-off #2: The setup is quite complex at first

To make contract tests work you need several days of work.

All teams need to understand the core concepts and how to write good tests with the good level of testing.
You need to add a pact broker in you environment (meaning you have to deal with security issues and keeping it running) this is mitigated by the fact the provided docker container works fine.
You need to make sure your providers applications run in your CI. By default there is no need to have a running instance in your CI but this is quite easy to do with docker.
You need to find a way to handle state in your provider apps to simulate all kinds of responses for instance you need to simulate your own dependencies to make the test reliable. Depending on the complexity of your dependencies (number of dependencies and number of possible outputs) this can be more or less difficult.

My Opinion

You can benefit of contract testing in many situations. Once you have overcome the initial cost to set up and understand how you should work with it for a big project it can even be a cost effective solution for smaller projects in your organization. So join the Pact users' community to improve your test stack and help your teams to ship faster. You are not the first, many companies like Atlassian use this tool already as they explain it in this video.

Go Further

To know if you can get benefit from Pact you should read this page of the documentation, in case of doubt read this. Then to learn how to use it you should begin here and here. To know where these tests should be used you can read the excellent article The Practical Test Pyramid which describes most kinds of test you can use in a project.

Appendix

If you want more details, here are some elements I used to create the evaluation table of the alternatives in the "For Whom?" part:

End-to-end testing

Pros:

When the tests is green you are sure everything works together since the tests checks everything at once

Cons:

The cost of maintaining an environment to test everything.
Test tend to be very flaky due to external causes.
You need to run all tests (and they are slow) before the real deployment of any part, which creates a congestion in your teams workflow.
The coverage of error cases costs even more because you need to set up your system to fail in expected ways and because each case is quite long to execute you tend to have few.
When you discover a real bug, you still need to work to find the root cause (slow to fix)

Use: When you want to check your most critic happy path and you have a way to have enough control on your system in an environment before prod (set up mocks for external services, set up well known data...)

Shared types

Pros:

Easy to set up

Cons:

You do not have the vision of what versions work together, this force to deploy the different pieces together which is a smell for down time.

Use: When you have a small enough system that you can deploy all at once and a technical way to share code between consumers and provider.

Versioned API without breaking change

Pros:

The problem cannot exist

Cons:

Maintenance cost is high, you either have to live with the burden of all your previous choices or find a way to know what is not used anymore (one way is a deprecation plan of older versions).
Risk of human error and still create a bug

Use: For public APIs this is a good solution as there is no way to contact each consumer directly.

Massive documentation and conception

this kind of massive

Pros:

Reduces the amount of bugs of any type.

Cons:

Forces you to make the perfect conception, keep track of every decision, to have comprehensive checklists and to train your whole team to follow all items.
Your process needs to be at the highest level, like the NASA for space shuttles.

Use: When lives depends on your software working perfectly

Contract testing

Pros:

Catch bugs between teams before deployment
Easy to maintain
Teams can deploy at their own pace
Documents expectations of all consumers and ensure they are met with no human intervention
Keeps track of what versions are compatible

Cons:

You need to carefully set up your workflow to integrate these tests.
You need to create tools to handle complex cases.
There is yet another tool to teach to your team.
You still need other kinds of tests to check the business rules of each application work as expected

Use: When you want to quickly develop a system with more than two interconnected applications (if you do not control the update of some applications like for mobile, each used versions count). And each application is developed by independent teams which communicate frequently. It can be better to have someone to train your team to reduce the setup cost.

Serverless Is More Than a Trend, Best Companies Adopt It

Mon, 29 Jun 2020 00:00:00 GMT

A recent article by the WSJ highlighting the benefits of serverless for organizations points at a building awareness by thought leaders about the impact this new trend can have for companies. It resonates with our own experience, celebrating successful serverless implementations with our clients in the last couple of years.

Yet, when pitching prospects about the value they can unlock through this technology, we often find that serverless is not very well known among decision-makers.

If the value is clear, why is there a lack of awareness?

The Term Serverless is Confusing

Ben Ellerby defended the serverless concept in an article, noting that it “is not a good term, yet it is used to describe a powerful and often misunderstood concept.” So, here is his definition:

When we first introduce people to serverless, there is this aha! moment when they understand it doesn’t mean there are no servers -- it’s just that they’re further outsourced to cloud providers.

This graphic positions serverless in the cloud landscape:

In a nutshell, using serverless gives you the ability to develop your own applications and microservices, to integrate off-the-shelf services like payment or user authentication, and to process events generated by these services or other SaaS. Meanwhile, the size of the engineering team decreases, as the full responsibility of managing the infrastructure is delegated to a cloud provider.

Serverless, Right for Large Companies as Well as Start-Ups

Gartner already mentioned serverless computing as one of the top 10 trends in 2019. Allied Market Research expects the global serverless architecture market to reach almost $22 billion in 2025, from $3 billion in 2017. Led by IT & Telecom, they expect the revenue to increase in all sectors, with the highest growth in the Media & Entertainment industry.

Large companies drove the market at its beginnings, with use cases such as real-time web applications and data processing; but SMEs will exhibit the highest growth rate between 2018 and 2025, according to the same study.

In his article Serverless is for Everyone, Sheen Brisals, Senior Engineering Manager at The LEGO Group, explains that “for a majority of start-ups, the tech stack is something they are born with”, therefore “change is easier when compared with a big enterprise”. But “starting small with serverless is perfectly applicable to a start-up as well as to a grown-up.”

As cloud providers invest in improving their solutions, more developers adopt serverless for developing microservices and functions, expanding the potential use cases.

Download the free presentation here

It’s Not Just About Saving Costs. Serverless Computing Solves Business Problems and Delivers Better Customer Experiences.

Reduced operational and development costs are the most important benefits sought by organizations going serverless, together with the automatic scaling of servers according to demand. On the product side, the increase in developer productivity and reduced engineering lead-time has a positive impact on the speed of delivery, thus on the time-to-market.

To overcome the challenges of dealing with legacy applications built during their 100+-year existence, companies like Liberty Mutual embraced serverless computing to solve real customers’ problems. Among the solutions that the IT team have built, Gillian Armstrong (Solutions Architect at Liberty Information Technologies) and Mark McCann (Architect) mention:

A virtual agent for their call center to answer the most common questions.
A virtual assistant to help the clients manage their policy documents.

According to Armstrong, with serverless:

The cost is so low, because it's all managed services, and they're able to bring it out, trial it with users for a little bit, make sure the whole system works, and then just scale seamlessly up.

At Theodo, we're also obsessed with business outcomes, and we train our engineers to focus on delivering value for our clients' end-users, in line with our Lean culture. When we start a new project, we recommend our clients use serverless components when we're confident it will add a competitive edge to the solution.

Check in this video to see how our ground transportation client used serverless for managing omnichannel, real-time notifications to passengers:

Serverless Computing Is as Sophisticated as It Is Powerful: the Main Roadblock is Training

According to the O’Reilly Serverless Survey 2019, with a panel of 1500 respondents from a wide range of locations, companies, and industries, some of the biggest challenges are linked to the education and hiring of staff.

In the organizations that have not (yet) adopted serverless, “fear of the unknown” comes in second after security concerns.

Some components proposed by Cloud providers, like DynamoDB from AWS, have huge potential, but using them correctly is still a skill that requires mastery.

To tackle these concerns, Theodo developed EventBridge Storming workshops, adapted from the Event Storming workshop technique by Alberto Brandolini. During these sessions, we work with business stakeholders to define a common language around application fields (e.g. order, payment, and shipping) that we can use to communicate with the team. We then use this vocabulary to design an architecture based on events grouped by domain and well-defined system boundaries.

Cloud providers and the user community also recognize the challenges faced in upskilling teams to handle serverless. The good news is that they are doing an outstanding job at improving the tools, developing best practices around security concerns, and sharing knowledge on novel ways to solve issues. Theodo is proud to have contributed to multiple frameworks and libraries in the space, an achievement recognized with a partnership with AWS and the Serverless Framework.

Conclusion

Reductions in Total Cost of Ownership and Time-to-Market are two compelling reasons to consider serverless for IT leaders. Besides, the granular nature of FaaS and managed services allow companies of all sizes to progressively build new applications and replace pieces of legacy architecture without requiring an engineering big-bang.

Chances are that you're already using serverless. If your technical stack includes a cloud provider such as AWS, Azure, or Google Cloud, your engineers have probably used their FaaS offerings for data processing or queue services for notifications. You're not starting from scratch!

This means that serverless is one of the easiest technologies to adopt. Even if your main infrastructure is still bare metal, you can experiment and iterate with serverless on a limited scope quickly and seamlessly. It will give you a better idea of the benefits and challenges of serverless for your company, right now.

Need help? Contact me at alexdb@theodo.com

Cover photo by Danielle MacInnes on Unsplash

Build an Animated Accordion List in React Native

Sat, 27 Jun 2020 00:00:00 GMT

Accordion lists (or ExpandableListViews) are now a material design staple and show up in quite a few apps. They are a great way to dynamically display information to a user.

When it comes to implementing one in your own app, sure, there are libraries that you can use. But implementing one yourself gives you greater control over styling, a smaller app bundle size, and it’s so easy it might even be faster! It’s also a great opportunity to sharpen your skills using React Native Hooks and the Animation API.

This is what we’ll be creating:

Just show me the code!

Alright, alright! Here’s a snack. Consider sticking around though - you might learn something.

Let’s get started

First off we need a functional component. This will take two arguments or props:

A string. The title in the top bar of the list item.
A ReactNode. The component in the expandable section.

For the second of these, React allows us to access child components via this.props.children in class components, or we can specify children as an argument to our functional component. Magic!

So our functional component has the form:

const AccordionListItem = ({ title, children }) => {
 ...
};
export default AccordionListItem;

And we can use it like so:

<AccordionListItem title={"List Item"}>
  <Text>Some body text!</Text>
</AccordionListItem>

The top bar of our list item will need to be wrapped in a TouchableWithoutFeedback since we want users to be able to open and close the list item by pressing it. This can contain the title we passed in via props.

<TouchableWithoutFeedback>
  <View style={styles.titleBackground}>
    <Text>{title}</Text>
  </View>
</TouchableWithoutFeedback>

Now let’s add a View for the list item body. This is the section that slides in and out when the header is clicked, and will contain our child component.

<View style={styles.bodyBackground}>
  {children}
</View>

Let’s see what we have so far:

Nothing revolutionary right? Bear with me, here’s where it gets interesting!

Now I’m hooked

Next up we’ll want to add a few React hooks to keep track of the state of our component. Here they are:

A state to keep track of whether the item is open or closed.
An Animated.Value to animate the opening and closing of the body section.
The height of the body section - this is what we will be changing in order to show and hide it.

Or in code form:

const [open, setOpen] = useState(false);
const animatedController = useRef(new Animated.Value(0)).current;
const [bodySectionHeight, setBodySectionHeight] = useState(0);

Note that we’re using useRef to create the animation object. Since our open state will persist across component refreshes, we want our animated value to do the same, otherwise they’ll get out of sync! useRef acts as a sort of state container for our Animated.Value, elevating it out of the component lifecycle.

Accordion time!

Now we need to tie our Animated.Value to the height of the body view. Our animated value can vary over whatever range we want but it’s easiest to pick a simple range like [0, 1]. Here, 0 maps to the list item being closed and 1 to it being open. As the animated value changes from 0 -> 1 we want the height of our view to go from 0 -> bodySectionHeight. To achieve this we can use the interpolate function:

const bodyHeight = animatedController.interpolate({
  inputRange: [0, 1],
  outputRange: [0, size.height],
});

Which simply maps the inputRange to the outputRange exactly as we described. When we change the value of the animatedController, the bodyHeight variable will scale this value relative to the height of our container. With all this set up, we can now write our toggleListItem() function, which will be called when we click on the item header.

We can think about what this function has to do as follows:

If the list item is open:
a. Change the animated value from 1 to 0.
b. Set the state to closed.
If the list item is closed:
a. Change the animated value from 0 to 1.
b. Set the state to open.

const toggleListItem = () => {
  if (open) {
    Animated.timing(animatedController, {
      duration: 300,
      toValue: 0,
    }).start();
  } else {
    Animated.timing(animatedController, {
      duration: 300,
      toValue: 1,
    }).start();
  }
  setOpen(!open);
};

I’ve set the duration to 300ms, but you can play around to find your preferred speed.

We’re getting pretty close. We’ll need to make our View an Animated.View in order to use React Native’s animation system, and then set the height of the animated view to be the bodyHeight variable.

<Animated.View style={[styles.bodyBackground, { height: bodyHeight }]}>
  {children}
</Animated.View>

Now for a sneaky step. We need to set an initial value for the height of our bodyView so we know what value to return when we open the list item. But, we can’t use the height of the animated view. This is because we are going to set the bodyHeight using the onLayout prop of View. For an animated view this is called many times while the view is animating! This is no good for us, we just want to set it once.

To fix this we can nest another regular view in our animated view which can have a constant height. We can then use this view’s height for the value of bodyHeight.

onLayout takes a function with an event parameter. We access the components height via event.nativeEvent.layout.height. So, putting it all together:

<Animated.View style={[styles.bodyBackground, { height: bodyHeight }]}>
  <View onLayout={setSize} style={styles.bodyContainer}>
    {children}
  </View>
</Animated.View>

Finally we need to set the overflow: “hidden” property on the bodyBackground style. This tells the children of the animated view what to do if they overflow the views box, which is exactly what will happen when the list item is closed. Setting this to hidden means they won’t be visible, perfect!

Realistic slide

What we have is nice, but it would be great if we could give the impression that the body of the list item is unfolding from the title bar. Fortunately there’s an easy way to achieve this! We can add the following properties to the bodyContainer style:

position: "absolute";
bottom: 0;

This ensures that the bottom of the inner view is always aligned with the bottom of the outer view, meaning the inner view should slide nicely up and down as the list item opens and closes.

Spinning arrows

Another nice feature would be to have an arrow indicator that rotates as the list item opens and closes. With our new knowledge of React Native animations, this should be easy! As our animated controller varies from 0 -> 1, we want the angle of our arrow to vary from 0 -> π radians or 0 -> 180° for regular folks. Using the same animated controller means it will always be in sync with the opening and closing. So we have:

const arrowAngle = animatedController.interpolate({
  inputRange: [0, 1],
  outputRange: ["0rad", `${Math.PI}rad`],
});

Then we can add our chosen arrow icon, within an Animated.View, into our title container view:

<View style={styles.titleBackground}>
  <View style={styles.titleContainer}>
    <Text style={styles.titleText}>{title}</Text>
    <Animated.View style={{ transform: [{ rotateZ: arrowScale }] }}>
      <DownArrowSVG width={20} height={28} />
    </Animated.View>
</View>

Here we have used the transform CSS property to pass in a z-angle of rotation for the view (in other words, the angle perpendicular to the z-axis - I had to work this out by trying rotateY and rotateX first!).

Take it Easing

One for the true perfectionists out there. React Native allows us to change the velocity of the animation via the easing parameter. This is a subtle change in our case, but it’s a good habit to get into as it can make a big difference in feel for some animations.

Most of the time I use the standard Material Design easing:

easing: Easing.bezier(0.4, 0.0, 0.2, 1);

Putting this in both our Animated.timing configs, we get our finished product!

Even after all that work our list item is less than 1kb in size once gzipped! I challenge you to find a library that’s smaller. We also have complete control over every aspect of our styling. Stick this in a FlatList component to create a nice scrollable list, or you can just stack them like regular components if you only need a few. Have fun with it!

Why it’s time to consider Azure to host your static application

Mon, 15 Jun 2020 00:00:00 GMT

The line between static & dynamic web applications is not what it used to be. With new frameworks like Gatsby or NextJS, the Jamstack is a strong option to get your application running in no time, whilst being able to scale it as needed.

When it comes to hosting your app, there are dozens of services available today, from Netlify to Firebase, but Microsoft has never really been one of those options (apart from hosting your app using an Azure Blob Storage) until last month, when they released their new service in preview, Azure Static Web Apps.

Here are the reasons why I think it is worth considering.

Get your app hosted & running in no time

If you haven’t used Azure before, you’d be surprised how easy it is to get started, and not only will you get set up in no time, but you’ll benefit from 12 months of Free Tier on most used services. Check it out here

Once you’ve picked your favourite frontend boilerplate and pushed it to Github, the only thing left to do is specify in your Azure Portal which branch you want to use. In less than a minute, your app will be hosted and available publicly.

In the background, Azure Static Web Apps service will generate & push a Github Action template to your repository, which will be triggered every time a branch is updated, creating for you multiple environments (even for each PR) without having to worry about CD. You can then enhance this action to add your own unit and integration tests.

Take advantage of the Microsoft Developer Experience

We have to admit that Microsoft have done an amazing job with VSCode. It is today, without any doubt, the most used IDE for web developers (haven’t tested it yet? check it out here https://code.visualstudio.com/ - In Theodo UK, we have decided to make it our default environment for web projects) And with such tools, the more popular they are, the more their ecosystem will become rich thanks to its community.

In our static web application example, it takes 2 clicks to download a LiveServer extension and have a server running your code locally, with hot reload. If you’re then planning to retrieve data in your app from an API, check out their extension which allows you to simulate your Azure Function running locally, test it, and deploy it all of that from your IDE.

Finally, you can write your app in TypeScript, framework designed by Microsoft, to type your JS code on your frontend and API, for a smooth and reliable experience.

Rely on Serverless to scale your app

If you’re not familiar with Serverless, we could summarise it this way: it is the concept of not provisioning servers, relying only on cloud services, to host your code and scale it up and down as needed automatically, greatly reducing the maintenance cost. (want to learn more? check out https://serverless-transformation.com)

As I mentioned just before, Azure Static Web Apps service allows you to use an API to retrieve data in your app. To do so, Microsoft give us the possibility to use Azure Functions, in JavaScript or TypeScript, to then be called directly from your frontend code. Once your environment variables defined, you just have to configure your SSL, your domain name and your app is good to scale!

Conclusion

Since Microsoft have gone down their open source journey, they are becoming a strong contender for a move to the cloud. Want to get started? Check out their quickstart

Don't overestimate the importance of performance when choosing a stack for your Web project

Thu, 14 May 2020 00:00:00 GMT

Understand how a suitable backend language can make a server handle higher loads and help reduce operating costs – or not.

A few months ago, a colleague of mine gave a talk detailing how great his recent experience with Go had been. As a side-note, he mentioned that Web applications written in Go were able to handle way greater loads than those powered by its mainstream Web competitors. This intrigued me: if this performance gap does exist, what is it due to, and is it significant enough for me, a Web developer, to actually care about it?

After a bit of research, I was confident I had a better understanding of the factors at play in the performance of Web servers. Still, I was lacking some quantitative insight that would allow me, at last, to decide whether taking these differences into account would help me bring more value to my next project. This is when I chose to run a small home-made benchmark comparing Node, PHP, and Go – three technologies we use at Theodo – that I will use to draw the final conclusions of this post.

What is performance anyway?

The first step of my investigation was to get sure of what I meant by performance. Although there is no single definition of this concept, I am confident that the following is relevant to my needs as a Web developer:

Performance is the ability of a server to handle a great number of requests during a given period of time.

Thus, a server that performs well according to this definition will be able to run on less powerful hardware, reducing operating costs and environmental footprint.

It is also correlated to other definitions of performance you may have thought about. For example, the faster a server can respond to a request, the more requests per minute it should be able to handle.

What a performant server is made of

The next thing to realize when trying to understand how a particular technology may influence the performance of a server is that during the whole process of serving an HTTP request, time dedicated by the CPU to executing code actually written by the Web developer or by the Web framework developers is often extremely small compared to:

Time spent performing heavy computations, such as hashing a password, manipulating images, etc.
Time spent waiting for network calls to terminate. Our server is most likely communicating with distant APIs and databases, or with the local file system. This takes way more time than executing a few thousand instructions on a CPU.

As counterintuitive as it may seem at first, heavy computations will not make much of a difference here. "Slower” interpreted languages will typically call well-optimized binaries under the hood to handle those kinds of tasks.

[Summary]:
   ticks  total  nonlib   name
      0    0.0%    0.0%  JavaScript
     72   63.7%  100.0%  C++
      1    0.9%    1.4%  GC
     41   36.3%          Shared libraries

What will make a difference on the other hand is the way the server uses the waiting times during the processing of a request to start handling other requests concurrently.

Scheduling how the CPU will handle numerous concurrent connections to the server is a complicated problem that can be addressed in a number of ways:

Typical PHP servers will let OS threads handle the task. To put it simply, threads are the mechanism that operating systems use to give us the feeling that they are processing several tasks in parallel. Thus, threads are also natural candidates for managing the concurrent processing of HTTP requests – each thread being responsible for a single connection – even though this is not what they were primarily designed for. This unfortunately comes with some performance issues: each new thread consumes quite a lot of memory, and time spent switching from the execution of a thread to another is also non-neglectable.
Technologies such as Node or Go will keep the processing of all the requests in a single or few threads but will use their own user-land mechanisms, such as an event-loop (Node) or Goroutines (Go), to address the scheduling issue. Although these two mechanisms will make Web developers write very different-looking code, they have in common that they do not suffer from the same overhead as OS threads do when opening a new connection or when switching from a connection to another.

Benchmark

We now know that thread-based Web servers should in theory be outshined by their counterparts that use a more suitable concurrency model. But is the performance gap wide enough to be of any importance to developers? Let’s find out!

To carry my little experiment, I set up a server and wrote three different routes in PHP, Nodejs, and Go (more details about versions and my exact protocol in appendix):

When route 1 is fetched, the server hashes a string then returns OK.
When route 2 is fetched, the server performs a call to a (very) distant server, waits for the response, then returns OK.
Route 3 is a mix between route 1 and route 2: when it is fetched, there is a 1% chance that the server hashes a string, else it performs a call to the distant server. It emulates a server that performs a lot of I/O (database queries for instance), and some CPU-intensive tasks (such as hashing passwords to identify users).

To compare performance, we’ll put some load on every route, one by one, for every language, and use the following comparison criterion:

Reference load = the maximum number of HTTP requests a server can handle in a minute, such as 95% of requests take less than 3 seconds.

Route 1 falls into the heavy computations category, so we expect to see little difference between all three technologies:

The small variations we observe can probably be explained by slight differences in bcrypt implementations. An interesting fact to notice here is that the reference load could have been predicted by dividing 60 seconds (1 minute) by the time it takes to complete a single isolated request (PHP case: 60 / 0.320 = 188). This means that our server is simply processing the requests one after the other, and has no opportunity to parallelize anything since there is no waiting time while handling a request.

So far, our results match the theory. Great!

Route 2 is much more interesting since there is now plenty of time for the server to parallelize requests while it is waiting for an API call to return. Let’s see what happens in this case:

We notice that the time it takes to handle a single isolated request is exactly the same no matter the language, which seems natural since the bottleneck of our route is a very long network call. However, huge differences arise when we start putting some load on the server:

Our Node server can handle about 20% more load than its PHP counterpart, which is significant, but definitely not as spectacular as the 1-to-4 ratio in RAM consumption. This seems to validate that thread-based servers are more memory-intensive than servers that use a more specialized concurrency model!
Meanwhile, our Go server is able to keep a low memory footprint while handling almost twice as many requests per minute than the former two. I’ll admit that I have yet to find a satisfactory explanation for this result, and if you do have one, I’d be very grateful if you could take the time to share it with me!

However, a real-world server could very well need to perform a few CPU-intensive operations among all these network calls, which is what I tried to emulate with route 3:

The most noticeable thing, in this case, is PHP’s memory consumption, and it is yet far from reaching any critical level. Reference loads on the other hand seem once again fairly similar.

Let's talk about money

All that is very interesting, but what matters in the end is how much money you will have to spend to run your application.

As an example, let's consider a large Symfony (PHP) project we are currently developing at Theodo. It handles about 50 million requests a day and runs on 32 machines with 8 cores and 32Gbytes of memory each. A similar infrastructure on AWS roughly costs $100,000 a year.

Our benchmark says that a more performant language could potentially divide infrastructure requirements by two. This would mean a $50,000 economy per year.

Conclusion

These results are telling me that choosing a suitable technology could have a measurable impact on operating costs if my service is mainly performing I/O, with close to zero CPU-intensive tasks. Otherwise, my application’s bottleneck will lie in the CPU-intensive code, and the technology I choose for writing my controllers will be of no importance whatsoever.

Let's also keep in mind that there are many more practical considerations that should be taken into account first when choosing a production-grade technology, as they may have a more significant impact on a project success. For example, a rich and mature ecosystem should come with a greater development speed, and should make it easier to find experienced developers to hire.

Note that Web performance in general should definitely be a concern for whoever wants to build a successful product! For that, you can check our performance experts at Theodo [FR].

Appendix: more details about the benchmark

The benchmark was run on an AWS EC2 t2 micro, Ubuntu 18.04.

Nginx 1.14.0, PHP-FPM 7.2
Node 12.14.0, managed by pm2 4.2.1 (and Nginx as a reverse proxy)
Go 1.13.4 (and Nginx as a reverse proxy)
I used loader.io as a load generator.
Code available here.

Measure the server-side impact of your application with PowerAPI

Thu, 14 May 2020 00:00:00 GMT

According to a report made by The Shift Project, the carbon footprint of the digital is estimated to exceed air travel with more than 4% of the total greenhouse gas emissions. This number is expected to double in the next five years!

Green IT principles aims to reduce the impact of the digital. As a developer, I wondered how these principles could be used for the web applications I worked on, and what could be improved.

I first looked at the definition of digital. It includes:

Networks (for instance wifi and 4G)
Data centers (used to host the servers and the data)
Terminals (smartphones, computers, TVs)
IoT (connected objects like a home assistant or a smart thermostat)

In this chart, the Shift Project shows that the energy consumption can be divided in two factors:

45% of the total emissions is dedicated to the production of digital equipment. To reduce the carbon footprint, it is important to keep a computer or a smartphone as long as possible. However, the average lifespan of a computer is estimated around 2 years. As developers, we can have an influence on the lifespan of our users' devices by making sure our applications are not too slow on old devices. Indeed, when you need 30 seconds to open most websites, your first thought usually is: "I really need to change my phone"!
55% of the emissions come from the usage of the equipment. We can have an impact on this energy consumption by designing sober and low-consuming applications.

So how can we measure the impact of a web application on our user’s device?

Some websites like Website Carbon ou EcoIndex (for french users) can estimate the client-side carbon footprint of a web application.
There is also the 1-byte-model which estimates the global carbon footprint. It takes into account the energy consumption of the data center, the network and the user device. A browser extension exists to see the impact of a website.

The data centers represents 19% of the digital energy consumption and some improvements in efficiency and performance can be found. However, there is no easy way to measure the carbon footprint of the server-side of an application. Therefore, it is complex to know where to begin to reduce the impact!

Example of a web application

At Theodo, we use an internal application to display the developers' current missions and availabilities. It is used by the sale representatives to see which developers are available and to staff them on new projects. One person has access to a CSV file online, and updates it several times a day. It is important to always have up-to-date data on the application in order to correctly staff the developers.

To update the application database, a script runs every 15 minutes: it imports the CSV file and parses it, then modifies the availability of the developers in the database. The file, which is quite large, is therefore imported 96 times a day. I had the intuition that it was consuming a lot of energy to run this script as many times, and I looked for a way to confirm or deny it.

I searched “How can I measure my server energy consumption” online and found out that simplest way was to plug a wattmeter into my computer, run the script locally and monitor the energy consumption. However, like most people, I don’t have a wattmeter at home, and I was not very keen on buying one for this unique usage.

I finally discovered a very useful tool: PowerAPI.

PowerAPI

PowerAPI is an open-source project developed by the Spirals research group (University of Lille 1 and Inria) in order to estimate the power consumption of a software, without any external device like a wattmeter. The calculation is made based on the analysis of the consumption of various hardware components (for instance CPU, memory, disk…). The objective is simple: design less energy-consuming software by easily monitoring their power consumption.

The power meter created by PowerAPI is composed of two components: the sensor and the formula

the sensor collects raw data from the hardware while the software is executed and stores it in a database.
the formula accesses this database, processes the raw data into a power consumption in Watt. The formula can store the results in the database or export it in a CSV.

As the sensor needs information from the Linux kernel, PowerAPI can only be used on Linux machines and not on a virtual machine.

Install PowerAPI

To install PowerAPI, you need:

Docker installed on your machine
a MongoDB database that can be accessed by your machine.

In my case, I decided to execute the project locally and deploy the MongoDB instance on my computer.

In the MongoDB database, I created a database (I decided to call it power-data) and two collections:

a collection to store the raw data from the sensor (in my example sensor-data)
a collection to store the computed results from the formula (formula-data)

You can either monitor the global power consumption of your machine, or monitor the power consumption of all Docker containers. As my project was using Vagrant, I decided to monitor the global power consumption of my computer. However, I recommend monitoring the Docker containers as it will reduce the “noise” created by your machine.

To install PowerAPI, the PowerAPI documentation will guide you. Once the MongoDB instance is created, the power meter can be deployed in two commands and a few minutes. Here’s a rapid recap, to measure the global power consumption:

Deploy the sensor: launch the first command to create the sensor Docker container.

docker run --privileged --net=host --name powerapi-sensor --privileged -td \
       -v /sys:/sys \
               -v /var/lib/docker/containers:/var/lib/docker/containers:ro \              
               -v /tmp/powerapi-sensor-reporting:/reporting powerapi/hwpc-sensor \        
               -n powerapi-sensor \
               -r "mongodb" -U "mongodb://ADDR" -D "power-data" -C "sensor-data" \
               -s "rapl" -o -e RAPL_ENERGY_PKG

Deploy the formula: launch the second command to create the formula Docker container.

docker run -td --net=host --name powerapi-formula powerapi/rapl-formula \
       -s \
       --input mongodb -u mongodb://ADDR -d power-data -c sensor-data \
       --output mongodb -u mongodb://ADDR -d power-data -c formula-data

The power meter is now ready, with two running Docker containers: the sensor and the formula.

Start measuring

If both Dockers are running, the power estimation is stored in the database. To stop monitoring, stop the Dockers.

The results

Once PowerAPI was deployed:

Step 1: I first started to monitor my computer energy consumption, to estimate what was the “noise” created by my machine.
Step 2: I launched the vagrant to start my application
Step 3: I ran the script 5 times in order to get average data.

The formula had stored all my power consumption data in the database. Now, I needed a way to display and analyse it.

How to process data from PowerAPI

First, I decided to use MongoDB Charts to display my results.

MongoDB provided me a simple interface to display the results written by the formula. I can for instance filter by date, to create a graph for a precise duration. On this chart, I displayed the energy consumption of my application during 1 minute.

However, in order to extract more information from the results, I decided to extract the data as a CSV, which can be done:

directly in the formula parameters by adding --output csv at the end of the command to launch the formula Docker
with a MongoDB command
via MongoDB Charts.

Once I had the data, I started to analyze it.

The results were given in Watts. When dealing with electric consumption, it is easier to use Watt-hours: for instance, you need 1 kWh to bake a cake in an oven during 1 hour. It means that your oven consumed 1000 Watts during 1 hour.

	Duration (s)	Average Power (W)	Consumption (Wh)
Step 1	225.5	0.81	0.05
Step 2	225.5	0.83	0.05
Import	225.5	5.47	0.34

The import lasted in average 225.5 seconds, about 3min45.

In Step 1, my computer alone consumed on average 0.81 W, i.e. 0.05 Wh during the imports.
In Step 2, my computer with the vagrant containing the application running used 0.83 W. Step 1 and Step 2 are almost equivalent in terms of Wh. In other words, running the application on Vagrant does not consume of lot of energy.
On average, one import used 0.34 Wh. The electric consumption almost did not vary between imports.

If I subtract the noise of the computer, an import alone consumes 0.29Wh. Imports are made 96 times a day, i.e. 35 064 times a year. It means the imports use each year around 10 kWh.

A Danish association, Tomorrow, created an interactive map to see in real time the countries' electricity production and its impact in terms of CO₂. Based on this website and on an 24-hours average, in France, 10 kWh corresponds to 3.5 kg of CO₂ emitted, as our energy comes mostly from nuclear. In the United States, 10 kWh corresponds to 11 kg CO₂.

If the server running my application was behaving exactly like my computer, it would use around 7 kWh a year.

To sum up, I estimated that the application was using 17 kWh each year, and 60% of this energy consumption is made by the imports. The rest is the energy consumes by the server, in the hypothesis of a physical server.

What can we usually do with 17 kWh? With 17 kWh, you can use your air-conditioning for 102 hours, cook 17 chickens in your oven or travel 8.5 km with a small electric car.

It is quite a lot for the server of a small web application.

What can be done to reduce the impact of this application?

I spoke with the developer and the person which was updating the CSV file, and we decided to:

take a quick action and rapidly stop importing the file outside work hours. Importing the file only between 9a.m. and 7p.m. on weekdays reduces the total consumption of the imports to 3 kWh a year, in other words a reduction of 41% of the electric consumption!
in the long term, create a button in the application, which can only be seen by the person updating the CSV file. He would use it when he updates the file, a few times a day. It will reduce the emissions of 95%.

What about the cloud?

Thanks to PowerAPI, I was able to estimate the impact of a costly feature on a physical server. I provided figures to help the project team make decisions to reduce the carbon footprint of the application.

However, this server runs on AWS, and PowerAPI cannot be used in a virtual environment in the cloud. In order to estimate the real impact, we need to find a methodology, using data from the cloud provider: for instance the amount of CPU used by the application.

I will try to see the real impact in the cloud of this feature in an upcoming article!

Flask Debugging in VS Code with Hot-Reload 🔥

Mon, 11 May 2020 00:00:00 GMT

I love using a debugger when I code. It allows me to quickly understand why something does not work as intended, but also to get a faster and deeper understanding of code I did not write.

Since I am so fond of using a debugger, when I started working on a Dockerized Flask application, my first online search was to find how to set one up for my application. But all the solutions I found had different flaws:

❌ The application port would be changed every time I started the application with the debugger
❌ Flask's wonderful hot-reload feature (the server restarting after saving changes to the code) was not supported
❌ The usage was clunky. To make the debugger work, I had to consistently add then remove multiple lines of work

So, I decided to craft my own debugger setup to fix all those problems.

Let's see how to setup a Dockerized Flask app with an efficient debugging flow 🎉!

🔧 Prerequisites

To follow this tutorial, you will only need the following installed:

VS Code
Docker
Docker Compose
The VS Code Python extension

Step 1: Docker setup

To follow this tutorial, make sure you have a Docker configuration similar to this:

# docker-compose.yml
version: "3.4"

services:
  flask-server:
    image: flask
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - 5000:5000
    volumes:
      - .:/app:cached

# Dockerfile
FROM python:3.8

EXPOSE 5000

# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE 1

# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED 1

# Install pip requirements
ADD requirements.txt .
RUN python -m pip install -r requirements.txt

WORKDIR /app

# Switch to a non-root user, please refer to https://aka.ms/vscode-docker-python-user-rights
RUN useradd appuser && chown -R appuser /app
USER appuser

Here we have setup a simple flask-server service that will run our Flask application inside a Docker container.

Step 2: Setup the debugger

VS Code configuration

The only configuration you will need is adding or modifying the .vscode/launch.json file:

// .vscode/launch.json
{
  "configurations": [
    {
      "name": "Python: Remote Attach",
      "type": "python",
      "request": "attach",
      "port": 10001,
      "host": "localhost",
      "pathMappings": [
        {
          "localRoot": "${workspaceFolder}",
          "remoteRoot": "/app"
        }
      ]
    }
  ]
}

Be sure to properly setup the pathMappings property. This will be used to link the files being executed in the Docker container to the files in your machine.
The .vscode folder stores all the project's VS Code configuration files.

Install the debugpy Python module

debugpy is a Python module that will allow you to spawn a debugger inside our Python code.

To install it, make sure to add debugpy to your requirements.txt:

flask==1.1.2
+debugpy

Not specifying the version (ie: not adding a line like this debugpy==1.0.0) will automatically install the latest version of the module.

Use debugpy to create a debug adapter instance

Create a debugger.py file in your application:

# debugger.py
from os import getenv

def initialize_flask_server_debugger_if_needed():
    if getenv("DEBUGGER") == "True":
        import multiprocessing

        if multiprocessing.current_process().pid > 1:
            import debugpy

            debugpy.listen(("0.0.0.0", 10001))
            print("⏳ VS Code debugger can now be attached, press F5 in VS Code ⏳", flush=True)
            debugpy.wait_for_client()
            print("🎉 VS Code debugger attached, enjoy debugging 🎉", flush=True)

Let's explain what is happening here:

Debug adapter logic

debugpy.listen(("0.0.0.0", 10001))

This will start the debug adapter that will listen for a client connection at the 0.0.0.0:10001 interface.

debugpy.wait_for_client()

This line will block program execution until a client (in our case, the client will be the VS Code debugger) is attached.

More logic for a better experience

if getenv("DEBUGGER") == "True":

We want the debugger to be spawned only if the DEBUGGER env variable is set to True. That way, we will still be able to run our application without it.

if multiprocessing.current_process().pid > 1:

In debug mode Flask uses a first process (with pid==1) to start child processes that handle connections. If the code below this line is executed by the main process, the debugging port is taken and subsequent child processes can't use the same port and are attributed a random port which prevents connections.

Instantiate the debugger

Given that your Flask application instance app is created in a file named app.py, call the initialize_flask_server_debugger_if_needed function inside it like this:

# app.py
from flask import Flask
from debugger import initialize_flask_server_debugger_if_needed

initialize_flask_server_debugger_if_needed()

app = Flask(__name__)

@app.route("/")
def home():
    return "Hello, Flask!"

Expose the debugging port

To be able to listen to the debug port of your Docker container, add the 10001 port to the list of exposed ports:

# docker-compose.yml
...
services:
  flask-server:
    ...
    ports:
      - 5000:5000
      - 10001:10001
    ...

Launch your application with a debugger 🎉

Everything is set up, now we only need to start our application!

Here are the docker-compose commands in a Makefile to launch your application with hot-reload using the flask executable.

# Makefile
## 🌶 flask and hot-reload
flask:
	docker-compose run --rm -e FLASK_APP=app.py -e FLASK_ENV=development --service-ports flask-server flask run --host 0.0.0.0

flaskdebug:
	docker-compose run --rm -e DEBUGGER=True -e FLASK_APP=app.py -e FLASK_ENV=development --service-ports flask-server flask run --host 0.0.0.0

The only difference between the two commands is the -e DEBUGGER=True parameter. This will set the DEBUGGER env variable to True inside your container, allowing the debugger code to be executed.

Here's a GIF to show the debugger in action:

To summarize, here are the steps to follow:

In a terminal, run make flaskdebug
When prompted ⏳ VS Code debugger can now be attached, press F5 in VS Code ⏳, press F5
Add a breakpoint to the line you want to debug by clicking left of its number
Call the corresponding route (localhost:5000/hello/flask in the GIF)
The debugger stops the code at your breakpoint, happy debugging 🎉

Hot-reload in action 🔥

Now that it's easy to use the debugger, let's see how the application hot-reload feature is supported.

As you can see in the GIF, after I saved modifications to a file:

The application restarts
The debugger is detached
I am prompted ⏳ VS Code debugger can now be attached, press F5 in VS Code ⏳ again
After I pressed F5, the application is now running with the new changes and the debugger is attached

Tips & tricks

Use the Debug Console to code on the fly

Whenever you are stopped at a breakpoint in your code, VS Code's Debug Console acts as a Python interactive console with the full current context of your code.

Here you can see that I was able to do the following:

Import date from the datetime module
Re-assign the name value to date(3000, 1, 1)
The route response is correctly updated with the modifications

The Debug Console is a very useful tool to quickly test edge cases and find fixes!

Running the app with gunicorn

To use gunicorn instead of flask to run the app, use these commands:

## 🦄 gunicorn and hot-reload
gunicorn:
	docker-compose run --rm --service-ports flask-server gunicorn --reload --bind 0.0.0.0:5000 app:app

gunicorndebug:
	docker-compose run --rm -e DEBUGGER=True --service-ports flask-server gunicorn --reload --bind 0.0.0.0:5000 --timeout 3600 app:app

Running the app as a top-level script

If you need to run your app as a top-level script, ie:

The app is run with python app.py

# app.py
...
if __name__ == "__main__":
  app.run("0.0.0.0", debug=True)

Be sure to make this modification:

if __name__ == "__main__":
    from debugger import initialize_flask_server_debugger_if_needed

    initialize_flask_server_debugger_if_needed()

    app.run("0.0.0.0", debug=True)

Thanks to this, if you want to use the app instance in another file:

# another-python-file.py
from app import app

The code initiating the debugger will not be executed when the import is made.

Wrap-up 🌯

I showed you how to resolve the three problems presented I encountered trying to setup a debugging flow in a Dockerized Flask application:

✔️ The application port does not change
✔️ Flask's wonderful hot-reload feature is supported
✔️ The debugger is easy to use

I created a repository containing a minimal application with everything shown in the article.

I hope that this article helped you setup a (better!) debugging flow of your Dockerized Flask application 💪.

You have a question? No problem, feel free to either contact me:

📧 by email
🐦 on twitter
🐙 by leaving an issue or a pull request on the tutorial repository
💬 by leaving a comment below

I will be very happy to discuss this tutorial with you 😃

Stop losing data when writing Django migrations !

Thu, 07 May 2020 00:00:00 GMT

Saying that database structure is important is sort of an obvious statement. That is of course if you decided to use a structured database technology. But in that case, you want your database structure to be the closest to your functional and business needs, and the tightest in order to be able to rely on that structure.

However, when I was working on Django projects, I sometimes found it painful to change the data structure when I needed to. However, I discovered a very useful tool offered by Django migrations : the RunPython command.

Simple migration

Let’s take a simple example and assume that we have a single model called Order inside an orders application.

We would then have the following models.py file:

from django.db import models


class Order(models.Model):
    reference = models.CharField(max_length=8)
    amount = models.DecimalField(max_digits=8, decimal_places=2)
    creation_date = models.DateField(auto_now=False, auto_now_add=False)
    due_date = models.DateField(auto_now=False, auto_now_add=False)
    customer_name = models.CharField(max_length=50)
    customer_address = models.CharField(max_length=50)
    customer_city = models.CharField(max_length=50)
    customer_zip_code = models.CharField(max_length=50)

    def __str__(self):
        self.reference

Running python manage.py makemigrations will produce the following migrations file, called 0001_initial.py :

from django.db import migrations, models

class Migration(migrations.Migration):

    initial = True

    dependencies = []

    operations = [
        migrations.CreateModel(
            name="Order",
            fields=[
                (
                    "id",
                    models.AutoField(
                        auto_created=True,
                        primary_key=True,
                        serialize=False,
                        verbose_name="ID",
                    ),
                ),
                ("reference", models.CharField(max_length=8)),
                ("amount", models.DecimalField(decimal_places=2, max_digits=8)),
                ("creation_date", models.DateField()),
                ("due_date", models.DateField()),
                ("customer_name", models.CharField(max_length=50)),
                ("customer_address", models.CharField(max_length=50)),
                ("customer_city", models.CharField(max_length=50)),
                ("customer_zip_code", models.CharField(max_length=50)),
            ],
        ),
    ]

Let's break down what's happening in this migrations file. The migration operation is represented as a python class, with 3 attributes:

initial states that this migration is the first of its Django app;
dependencies list all the migrations that need to be applied before this one can be applied. Here it is empty since the migration is the first of its Django app;
operations is the most important part. It is a list of actions that Django will apply to the database. Here the database was empty, so we created the model from scratch with the CreateModel method.

The python manage.py makemigrations performs well when the operations to perform are simple enough. Renaming a model, or renaming a field will be treated as equivalent database operations for example.

After applying the migration, let's create a few entities in the database.

Complex structure migration

Let's start again from our previous example and observe something that you may already have noticed. The 4 fields customer_name, customer_address, customer_city, and customer_zip_code are functionally attached to the Order entity, while they represent another physical entity, a customer. And that is fine as long as there is no duplication. But now let's imagine that for business purposes, you need to abstract a Customer entity to which the Order entities are linked.

Your models.py file would then be:

from django.db import models

class Customer(models.Model):
    name = models.CharField(max_length=50)
    address = models.CharField(max_length=50)
    city = models.CharField(max_length=50)
    zip_code = models.CharField(max_length=50)

    def __str__(self):
        return self.name

class Order(models.Model):
    reference = models.CharField(max_length=8)
    amount = models.DecimalField(max_digits=8, decimal_places=2)
    creation_date = models.DateField(auto_now=False, auto_now_add=False)
    due_date = models.DateField(auto_now=False, auto_now_add=False)
    customer = models.ForeignKey(Customer, on_delete=models.PROTECT)

    def __str__(self):
        return self.reference

This configuration now fits our functional needs. Let's try and generate the migration !

> You are trying to add a non-nullable field 'customer' to order without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
 2) Quit, and let me add a default in models.py

There is a problem however. Django recognizes that we are trying to create a customer field on the Order model, which can't be None but is not defined. So it wants to add a non-null value to all the existing entries in the database, and asks us if we want to provide it now or want to set it in the Order model.

However, none of these options satisfy us. We don't want our orders to have customers that are either None or a default customer, which would lose all the existing data.

Granted, we could also execute this migration, then manually set the customer attribute of all the Order model, but then this forces you to execute this same action on all of your environments. That would also make the migration quite difficult to rollback.

Fortunately, Django comes with an built-in solution to deal with this limitation.

Generating a custom migration

First of, let's start by generating an empty migration that we will then edit.

> python manage.py makemigrations orders --empty

The generated migration file will look like this

from django.db import migrations

class Migration(migrations.Migration):

    dependencies = [
        ('orders', '0001_initial'),
    ]

    operations = [
    ]

We then need to build our custom migration. It needs to have 6 steps :

Create the new Customer model in the database, with all the necessary fields.
Create a nullable customer foreign key on Order
Set the Order fields to transfer as nullable
Transfer the data from the Order model to the Customer model.
Set the new customer field on Order as non-nullable
Remove the old fields from the Order model.

The Django documentation explains all the database operations needed.

Let's write this migration :

from django.db import migrations, models
import django.db.models.deletion

class Migration(migrations.Migration):

    dependencies = [
        ("orders", "0001_initial"),
    ]

    operations = [
        # step 1: add the new Customer model
        migrations.CreateModel(
            name="Customer",
            fields=[
                (
                    "id",
                    models.AutoField(
                        auto_created=True,
                        primary_key=True,
                        serialize=False,
                        verbose_name="ID",
                    ),
                ),
                ("name", models.CharField(max_length=50)),
                ("address", models.CharField(max_length=50)),
                ("city", models.CharField(max_length=50)),
                ("zip_code", models.CharField(max_length=50)),
            ],
        ),

        # step 2: add the nullable foreign key field `customer` to Order
        migrations.AddField(
            model_name="order",
            name="customer",
            field=models.ForeignKey(
                null=True,
                on_delete=django.db.models.deletion.PROTECT,
                to="orders.Customer",
            ),
        ),

        # step 3: set the order fields as nullable
        migrations.AlterField(
            model_name="order",
            name="customer_address",
            field=models.CharField(null=True, max_length=50),
        ),
        migrations.AlterField(
            model_name="order",
            name="customer_city",
            field=models.CharField(null=True, max_length=50),
        ),
        migrations.AlterField(
            model_name="order",
            name="customer_name",
            field=models.CharField(null=True, max_length=50),
        ),
        migrations.AlterField(
            model_name="order",
            name="customer_zip_code",
            field=models.CharField(null=True, max_length=50),
        ),

        # step 4: transfer data from Order to Customer
        ...

        # step 5: set the `customer` field as non-nullable
        migrations.AlterField(
            model_name="order",
            name="customer",
            field=models.ForeignKey(
                null=False,
                on_delete=django.db.models.deletion.PROTECT,
                to="orders.Customer",
            ),
        ),

        # step 6: remove the old Order fields
        migrations.RemoveField(model_name="order", name="customer_address",),
        migrations.RemoveField(model_name="order", name="customer_city",),
        migrations.RemoveField(model_name="order", name="customer_name",),
        migrations.RemoveField(model_name="order", name="customer_zip_code",),
    ]

In this migration, step 2 and 3 basically loosen the data structure, preparing it for the data transfer. Then step 5 and 6 tighten it again !

Let's now dive in step 4, where the magic happens !

Write the data transfer function

In order to perform this operation, we are going to use the RunPython migration operation. What it basically does is execute a python function using the ORM.

Its syntax is the following :

# step 4: transfer data from Order to Customer
migrations.RunPython(order_to_customer, reverse_code=customer_to_order)

In this step of the migration, we specify two functions :

order_to_customer will be run when running the migration ;
customer_to_order will be run when reversing the migration.

def order_to_customer(apps, schema_editor):
    Order = apps.get_model("orders", "Order")
    Customer = apps.get_model("orders", "Customer")
    for order in Order.objects.all():
        customer, _ = Customer.objects.get_or_create(
            name=order.customer_name,
            address=order.customer_address,
            city=order.customer_city,
            zip_code=order.customer_zip_code,
        )
        order.customer = customer
        order.save()

def customer_to_order(apps, schema_editor):
    Order = apps.get_model("orders", "Order")
    for order in Order.objects.all():
        order.customer_name = order.customer.name
        order.customer_address = order.customer.address
        order.customer_city = order.customer.city
        order.customer_zip_code = order.customer.zip_code
        order.save()

These functions look a lot like what you would write to manually transfer the data from one model to the other. However there is one trick. It isn't possible to import the models normally with from orders.models import Order, Customer. Instead, we need to use a versioned model passed to the migration function by Django.

Running the migration

After running this migration via python manage.py migrate, we can check that the migration has created the Customer instances and linked them to the correct Orders.

Customers

Orders

Et voilà ! This migration is now operational. It can also be easily rolled back with python manage.py migrate orders <previous_migration_id>. The data will then be transferred back to the Order model.

Conclusion

You now have a way to stop worrying about losing data when migrating your database with Django! The Django ORM is a great tool and although it does the job most of the time, understanding what happens under the hood is a great way of making your life easier when dealing with migrations.

Sources

Django documentation on migrations:
- https://docs.djangoproject.com/en/3.0/howto/writing-migrations/
- https://docs.djangoproject.com/en/3.0/ref/migration-operations/
Link to the Github repo with the example https://github.com/fargito/django-runpyton

How to Remain Agile with DynamoDB

Tue, 05 May 2020 00:00:00 GMT

Amazon DynamoDB is built to deliver single-digit millisecond performance at any scale. It is built to store Terabytes of data. It is built to support Amazon's Cyber Monday traffic.

However, this scalability comes with an overhead. No matter what resource you look up, you will be constantly reminded that "You Must Know Your Access Patterns In Advance". Amazon Marketplace had been running for years before they migrated to DynamoDB so they were well equipped to know how they needed to access their data and fully reap the rewards of a NoSQL setup.

In contrast, a new startup with an exciting new product wants to build an MVP quickly. They don't know how their app will grow over time. They want to be agile and react to their users to develop their product accordingly. They can't know all their access patterns in advance.

This no doubt poses an issue, but not an insurmountable one.

Being able to map your access patterns in as much detail as possible will translate to a smooth DynamoDB adventure. However, building a startup is about disrupting the market so you can't be afraid of a little turbulence along to way to reach your goal.

If you use DynamoDB from the start you will never have to worry about the scalability of your product. If your app gets a massive social media spike, you won't have to scramble to prevent your site going down. And reaching that scale is what all startups aspire to so how can we build towards this while limiting the turbulence?

Check out my full article on our Serverless Transformation Blog.

Why You Need a Framework When Doing Serverless

Mon, 27 Apr 2020 00:00:00 GMT

We are proud to have been selected as a Serverless Framework official partner. It rewards our investment in the technology and reflects our strong belief that this framework is currently the best in the market.

Using a Framework When Going Serverless is a Must

Recently, one of our clients’ technical advisors shared with us the challenges that he had encountered when testing serverless a few years ago. As he started to use AWS Lambda functions, he quickly lacked visibility of the infrastructure design and struggled to figure out which version of the code was running in production. Fortunately, these issues are now solved by frameworks for serverless applications.

At Theodo, we began leveraging serverless functions a few years ago for projects requiring asynchronous workloads. To avoid the aforementioned pitfalls, we looked into tools that would allow us to be more efficient in managing our serverless projects. The most popular tool was, and remains today, the Serverless Framework.

The main advantage of the Serverless Framework is to allow developers to write to one single repository containing:

The code of the business logic run by their serverless functions,
All the files that describe their application’s infrastructure, including:
- Database services like DynamoDB,
- Authentication services like Cognito or Auth0,
- Notification services like AWS Simple Notification Service or Twilio,
The scripts they need to easily deploy their application in one command to any environment,
Tools allowing developers to emulate services like DynamoDB in their local development environment. Other things that the Serverless Framework achieves is the deployment of a new stack with the whole infrastructure in a single command, allowing to run automated tests consistently on a ephemeral — hence cheap — production-like environment. This further stresses the productivity gains and cost reductions observed by companies embracing serverless.

Why We Chose to Work With the Serverless Framework

Its Large Community Guarantees its Relevance

The suite of serverless services offered by cloud providers is constantly expanding. Therefore, the value of a serverless application framework lies in the ability of its community to stay current.

The Serverless Framework has the largest community among its competitors, and it provides many plugins that gradually add support for new services as they are released. You can read about a few of our favorite tools for this framework in this article.

Its Independence From Cloud Providers

Other frameworks, like AWS’ own Serverless Application Model (or SAM), are open-sourced and maintained by cloud providers themselves — offering early support for their newest features and services. The purpose of some third-party frameworks, however, is to enable developers to deploy their serverless applications to any cloud provider. As the concerns about vendor lock-in grow, the appeal of using a cloud-agnostic framework becomes stronger. It allows you to move your infrastructure to another cloud provider if necessary.

The development of a reliable and powerful framework is a daunting and ambitious task, but we believe in the purpose of the Serverless Framework and we enjoy using it daily on many of our projects. We contribute to the community, for example with sls-dev-tools, our open-source project that gives developers visibility on their serverless application directly from the command line.

With close to sixteen million downloads and hundreds of thousands of weekly deployments, Serverless Framework has become the benchmark in the serverless ecosystem — and this is only the beginning.

Want to know more about how Serverless can help your company? I'd be happy to help. Please reach out here!

Reduce redux boilerplate with redux-toolkit

Mon, 20 Apr 2020 00:00:00 GMT

You're looking for an easier way to write your reducer, aren't you ?

When writing plain redux, you have to define an action, an actionCreator and a reducer to update the store of your app. The reducer is in most cases a big switch to handle all the actions dispatched by actionCreators in your app. This leads to a lot of boilerplate and make the code more difficult to understand. There are many librairies out there aiming to reduce this boilerplate.

At Theodo, we used typesafe-actions to reduce redux boilerplate. The redux team recently recommended @redux-toolkit as the "official recommended approach for writing Redux logic". Thus, I tested it and wrote this article to show you the advantages of such librairies through a simple example.

The example

The example consist in changing the owner of a car :

There is one car and 2 possible owners ("JOHN" or "JANE").

When you click on one of the owner, the car owner in the redux store is changed to match the owner clicked.

Plain redux (28 lines)

With plain redux as on the code below, there are :

an action (UPDATE_OWNER),
an actionCreator (updateOwner)
a reducer (itemReducer) to handle all the actions (codesandbox).

The first simplification external librairies provide is to get rid of the action and avoid the switch in the reducer function.

Both typesafe-actions and redux/toolkit give you two functions to simplify action and reducer : createAction and createReducer

We do not need to keep the UPDATE_OWNER action type anymore in an variable to handle it in the reducer.

The switch disapears

Code is easier to read and understand

Typesafe action (21 lines ~ -25%)

With createAction and createReducer functions there is no action constant UPDATE_OWNER and no verbose switch in our reducer anymore : the reducer reacts directly to the actionCreator updateOwner (codesandbox)

Type-safe action provides also a clearer way to write the reducer using the chain API (codesandbox)

You can even combine actions easily and/or chain action handlers:

const counterReducer = createReducer(0)
  .handleAction([add, increment], (state, action) =>
    state + (action.type === 'ADD' ? action.payload : 1)
  )
  .handleAction(add, (state=> action) => ...);

The code is clearer and easier to understand

Remarks :
- @reduxjs/toolkit provides also a `createAction` function but used in a slightly different manner see here.
- @reduxjs/toolkit `createReducer` function is used as typesafe-action

@redux/toolkit (18 lines ~ -35%)

@reduxjs/toolkit goes further and provides a function createSlice, allowing actions and reducers to be created in the same function. It's very powerfull, give it a try codesandbox.

Moreover Redux toolkit is shipped with ImmerJS, "a tiny package that allows you to work with immutable state in a more convenient way".

No need to duplicate the state with spread operators everywhere: You can directly update the state and ImmerJs will take into account the changes and create a new state from the precedent state + the changes you made. Have a look again to the reducer function 👇🏽

const { actions, reducer: itemReducer } = createSlice({
  name: "items",
  initialState: initialItemState,
  reducers: {
    updateOwner: (state: ItemState, { payload: { user, item } }: UpdateOwnerAction) => {
      // This long code with many ... is replaced by one line thks to ImmerJS
      // return {
      // ...state,
      // [item.id]: { ...state[item.id], owner: user }
      // }
      state[item.id].owner = user;
    }
  }
});

This makes reducer and actions easier to understand. Goodbye "desctructuration ..." 😍.

⚠️ Redux teams recommends to read ImmerJs pitfalls before using it, so read it 😉

In the end of March 2020, the redux toolkit team added 2 api functions (createEntityAdapter & createAsyncThunk) to simplify reducer logic.

In the last example I use the createEntityAdapter function : the entityAdapter "generates a set of prebuilt reducers and selectors for performing CRUD operations on a normalized state structure."

After normalizing my itemState and creating the itemsAdapter, I can simplify my reducer function to use the generic entityAdapter.updateOne to handle the ownerUpdate action (check full code here).

This gives you a very fast way to write your reducer logic.

Have a look at redux-toolkit doc to fully understand the createEntityAdapter API

Conclusion :

Reasons to use typesafe-actions

If you have a lot of actions to handle and one reducer has to react to several actions then typesafe-actions chaining will be right for you

⚠️ Typesafe-actions maintainer is currently updating their API to simpllify the syntax. If you want to contribute. ⚠️

Reasons to use redux/toolkit

Easier to write and understand reducers with ImmerJs
Slicers really concise
The API encourages you to use a normalized state and provides you with practical CRUD function to update data in your store

As redux/toolkit is the recommended library by the redux team, you should give it a try. Slicers + ImmerJS = 🔥 : your code is shorter and easier to understand. Moreover the createEntityAdapter will help you to normalize your state and write a new reducer faster thanks to the many CRUD functions provided.

Demystifying Building Native Modules for React Native

Fri, 17 Apr 2020 00:00:00 GMT

You’ve just released a new native SDK. Could you bring more business to your company by releasing on a cross-platform development framework?

Maybe you’re a developer, longing for a native module of an open-source platform API (or even the third-party SDK above 🙄) to be released. Could you fast-track the development yourself?

You could build your own module. With React Native, this becomes a straightforward process.

Let’s talk about React Native for a second.

Why React Native

For those unfamiliar with React Native: it is a cross-platform development framework allowing you to create native apps for iOS and Android. Using React, you can maintain two platforms using the same codebase, allowing faster development, iteration and knowledge-sharing.

With this framework, we have two sides; JavaScript and Native. Between the two is a bridge, allowing bidirectional and asynchronous communication. This is the power of React Native, on top of a multitude of other benefits.

In 2020, we see the likes of Facebook, Bloomberg and Shopify [1] using React Native to develop their mobile applications, amongst others in the Fortune 500. With over 11 million Javascript developers [2] and more companies switching to React Native, releasing your technology for React Native could bring more growth.

We’ve seen other cross-platform development frameworks like Flutter and Xamarin climb the ranks over the last few years, but React Native is still gaining popularity and ever-increasing its performance as it gets more mature.

So hopefully that's convinced you to consider releasing a Native Modules for React Native if you're releasing Native SDKs. Let’s break down how you’d do that.

Native to React Native

As previously mentioned, communication from the native world to React Native is asynchronous. What this means is that any values must be sent through asynchronous callbacks, promises and events, each sent on a batched message queue. This is the architecture as of April 2020; there is currently a re-architecture of the React Native internals and due to be released mid-2020 [3].

On the native side:

The main thread is responsible for the UI.
The shadow queue is responsible for layout calculations.
Each native module has its own thread (Android shares a thread pool)

On the JavaScript side:

JavaScript VM thread which runs the bundled JS code and sends instructions to the native threads via the bridge.

The two sides communicate over the bridge using the message queue.

JavaScript knows about your native modules at runtime - there’s a JSON representation of each Native Module (consisting of module id, method id and arguments) and we can call methods on the Native Module this way.

For the other direction, Native to JavaScript, we can use promises, callbacks and event to transfer data.

To read more about the React Native internals and the upcoming re-architecture, read this article [4].

Module structure

To get started building your native module, there are a number of tools to help set up the skeleton of your project. For example:

We like Bob, as your library will then come pre-configured with TypeScript and support for Kotlin and Swift. If you've got any other favourites, let us know!

For project structure, if you’ve chosen Bob, the library should now have the following project structure:

- /ios
- /android
- /src
    - index.tsx
- /yourExampleApp
    - package.json
    - ...
- ...

In the iOS and Android folders, you’ll want to have your native SDK code - either available in your project or using git submodules to keep the version control.

Exposing iOS Modules

On the iOS Side, there are two languages at play: Objective-C and Swift. You can write the native module entirely in Objective-C, or write methods in Swift and expose them to Objective-C by using the @objc attribute.

The steps are then as follows:

Create a Bridging Header file (Objective-C, .h file) - Here you'll import:

#import "React/RCTBridgeModule.h"
#import <React/RCTLog.h>
#import <React/RCTEventEmitter.h>
#import <React/RCTConvert.h>

Create a Bridging Command file (Objective-C, .m file), expose modules with RCT_EXPORT_MODULE and Expose methods with RCT_EXPORT_METHOD

#import <React/RCTBridgeModule.h>
#import <React/RCTEventEmitter.h>
#import "MyModule.h"

@interface RCT_EXTERN_MODULE(MyModule, NSObject)

RCT_EXTERN_METHOD(myMethodWithNoParams)

RCT_EXTERN_METHOD(myMethodWithAPromise:(NSString)input
resolver:(RCTPromiseResolveBlock)resolve
rejecter:(RCTPromiseRejectBlock)reject)

+ (BOOL)requiresMainQueueSetup
{
    return YES;
}

@end

Define event emitters, implementing the Delegate pattern.

#import <React/RCTBridgeModule.h>
#import <React/RCTEventEmitter.h>
#import <React/RCTConvert.h>

@interface RCT_EXTERN_MODULE(MyModuleEmitter, RCTEventEmitter)

// Create a singleton for the EventEmitter class

- (id)allocWithZone:(NSZone *)zone {
 static MyModuleEventEmitter *shared = nil;
 static dispatch_once_t onceToken;
 dispatch_once(&onceToken, ^{
 shared = [super allocWithZone:zone];
 });
 return shared;
 }

- (BOOL)requiresMainQueueSetup
 {
 return YES;
 }

@end

For Swift, a good pattern to use is the Observer-Command-Emitter pattern:

This ties together your observer which implements the delegate, the event emitter which triggers events and the commands which will be the methods to expose to JavaScript. In the above code snippets, you've seen the Objective-C .m files for the above.

Exposing Android Modules

The steps for Android are similar:

Create a ReactPackage to declare the modules to expose:

package com.mymodule

import java.util.Arrays

import com.facebook.react.ReactPackage
import com.facebook.react.bridge.NativeModule
import com.facebook.react.bridge.ReactApplicationContext
import com.facebook.react.uimanager.ViewManager

class MyPackage : ReactPackage {
override fun createNativeModules(reactContext: ReactApplicationContext): List<NativeModule> {
return Arrays.asList<NativeModule>(MyModule(reactContext))
}

   override fun createViewManagers(reactContext: ReactApplicationContext): List<ViewManager<*, *>> {
       return emptyList<ViewManager<*, *>>()
   }

}

Create your module extending ReactContextBaseJavaModule to declare the methods to expose using @ReactMethod . Implement listeners to emit events.

package com.mypackage

import com.facebook.react.bridge._
import com.facebook.react.modules.core.DeviceEventManagerModule.RCTDeviceEventEmitter
import java.util.\*

enum class MyModuleError(val errorCode: String, val message: String) {
 INPUT_NOT_RECOGNISED( "INPUT_NOT_RECOGNISED", "Input was not of the correct format."),
}

class MyModule(reactContext: ReactApplicationContext) : ReactContextBaseJavaModule(reactContext), MyListener {
 companion object {
   const val MODULE_NAME = "MyModule"
 }

 override fun getName(): String {
   return MODULE_NAME
 }

 private fun createErrorHandler(promise: Promise): (TokenError) -> Unit {
   return fun(error: TokenError) {
     promise.reject(error.errorCode, error.message)
   }
 }

 @ReactMethod
 fun myMethodWithNoParams() {
    doSomething();
 }

 @ReactMethod
 fun myMethodWithAPromise(input: String, promise: Promise) {
   val handleError = createErrorHandler(promise)

   doSomethingWithInputSucceeds(input)?. let {
     promise.resolve("MY_METHOD_SUCCESS")
   } ?: kotlin.run {
     handleError(ModuleError.INPUT_NOT_RECOGNISED)
   }
 }

 override fun onEventSuccess(result: String, message: String) {
   val body: WritableMap = Arguments.makeNativeMap(mapOf(
     "result" to result,
     "message" to message
   ))
   this.reactApplicationContext.getJSModule(RCTDeviceEventEmitter::class.java)
     .emit("MyModule/eventSucceeded", body)
 }

 override fun onEventFailure(error: String, message: String) {
   val body: WritableMap = Arguments.makeNativeMap(mapOf(
     "error" to error,
     "message" to message
   ))
   this.reactApplicationContext.getJSModule(RCTDeviceEventEmitter::class.java)
     .emit("MyModule/eventFailed", body)
 }
}

Your new module

Now you’ve exposed your native modules and methods to React Native and can continue to build on top of them. What’s left is to import them from NativeModules from react-native and you’re ready to create your app:

In /src of your Native Module:

import {
  NativeModules,
  DeviceEventEmitter,
  NativeEventEmitter,
} from "react-native";

export const { MyModule, MyModuleEventEmitter } = NativeModules;

Now you're ready to go. Just yarn add react-native-myModule and import the above MyModule and emitters to get going.

To launch your development experience and quality to the next level:

Add a React Native deployment pipeline - https://blog.theodo.com/2019/04/react-native-deployment-pipeline/
Add end-to-end tests with Detox - https://github.com/wix/Detox
Add support for offline - https://dev.to/reactnativeradio/rnr-157-building-great-offline-ready-apps-in-react-native-with-josh-warwick

What a great development set up!

Summary

Once that’s all ready, your module is tested, you’re ready to publish. Very straightforward with npm publish[5].

There you have it. A native module which can now be released to the JavaScript tech community - perhaps bringing in more customers for your business or fast-tracking your next development track.

Hopefully this article gave you an overall idea of where to start building your bridge module. If you have any questions, let us know, we'd be happy to help!

For more resources, the following are some good resources:

Sources + links:

How to convert an imperative library to a declarative React component: the Sketchfab Viewer

Fri, 10 Apr 2020 00:00:00 GMT

I love using React for multiple reasons. I like the writing style, I like the tooling, I like the ecosystem.

But sometimes you run into a library that doesn't have a React adapter. It can either be pretty old, kind of niche, closed source, etc.

In this series of articles, we’ll look into how to convert an existing imperative JS library into a declarative React component.

We’ll use the Sketchfab Viewer as an example. Sketchfab develops an amazing 3d suite that includes a very configurable web viewer for use with their models.

You can see that their doc is pretty extensive, if a bit messy. It includes a lot of examples and it took me a long time to run into a use case that was not already covered. Yet you can also see that they use an imperative style of JS that doesn’t fit well into the React patterns.

api.addEventListener("viewerready", function () {
  console.log("Viewer is ready");
});

We are going to build a React function component on top of that library. It will accept a config object to declaratively change a color on the model. It will also expose a callback to react to a click event on a part of the model. And we'll use hooks because they're fun!

You can find the final code for this part here.

Part 1: Initializing and exposing the api object
Part 2: Working on it!

Initializing and exposing the api object

Throughout this project we'll work with this fancy chair, courtesy of the official Sketchfab examples.

Start the project with create-react-app

First things first, let's generate a basic CRA project:

npx create-react-app sketchfab-react

In src/, add a Viewer.jsx file and create an empty component inside.

import React from "react";

export const Viewer = () => <></>;

Add that component inside App.jsx

import React from "react";
import "./App.css";
import { Viewer } from "./Viewer";

function App() {
  return (
    <div className="App">
      <Viewer />
    </div>
  );
}

export default App;

Now let's import the library. Sketchfab provides a package install through npm but that's not always the case for other libraries, so we'll include the source in the head of public/index.html.

<script
  type="text/javascript"
  src="https://static.sketchfab.com/api/sketchfab-viewer-1.7.1.js"
></script>

This import will make window.Sketchfab available globally. If you're using Typescript, you will need to ignore it.

Render the Sketchfab Viewer in an iframe

According to the docs, we need to create an iframe and give it to the Sketchfab constructor to render the Viewer.

The way to do this in React is, in Viewer.jsx:

import React, { useEffect, useRef } from "react";

// Our wonderful chair model
const MODEL_UID = "c632823b6c204797bd9b95dbd9f53a06";

export const Viewer = () => {
  // This ref will contain the actual iframe object
  const viewerIframeRef = useRef(null);

  const ViewerIframe = (
    <iframe
      // We feed the ref to the iframe component to get the underlying DOM object
      ref={viewerIframeRef}
      title="sketchfab-viewer"
      style={{ height: 400, width: 600 }}
    />
  );

  useEffect(
    () => {
      // Initialize the viewer
      let client = new window.Sketchfab(viewerIframeRef.current);
      client.init(MODEL_UID, {
        success: () => {},
        error: () => {
          console.log("Viewer error");
        },
      });
    },
    // We only want to initialize the viewer on first load, so we don't add any dependencies to useEffect
    []
  );

  return ViewerIframe;
};

And you should have the viewer displaying!

Expose the api object to access it imperatively

One of the great things about the Viewer API is that it allows us to transform our model from the browser. Here, we'll try to make the chair red.

First we need to extract the api object and store it. We could put it in the state of our Viewer component, but we'll want to use it higher up in the tree, so we will pass a ref as prop and give it the api.

export const Viewer = ({ apiRef }) => {
	// ...
        success: api => {
          // Store the initialized api inside the provided ref
          apiRef.current = api;
        },
	// ...
};

Now, let's get it inside App.

function App() {
  const apiRef = useRef(null);

  return (
    <div className="App">
      <Viewer apiRef={apiRef} />
    </div>
  );
}

To check that it works, let's try to change the color of the background. There's a method for that!

Let's add this code to App.jsx:

function App() {
  // ...
  const changeBackgroundColor = () => {
    apiRef.current.setBackground({
      color: [Math.random(), Math.random(), Math.random(), 1],
    });
  };

  return (
    <div className="App">
      <button onClick={changeBackgroundColor}>Change background color</button>
      <Viewer apiRef={apiRef} />
    </div>
  );
}

Save, click on the button, voilà!

If it doesn't work, check that you have clicked on the play button of the viewer beforehand to load the model ;)

Change the chair color

Alright, that function is a bit bigger but you'll have to trust me.

function App() {
  // ...

  const changeChairColor = () => {
    apiRef.current.getMaterialList((err, materials) => {
      const plasticMaterial = materials.find(
        (material) => material.name === "Blue plastic"
      );
      plasticMaterial.channels.AlbedoPBR.color = [
        Math.random(),
        Math.random(),
        Math.random(),
      ];
      apiRef.current.setMaterial(plasticMaterial, () => {
        console.log("Updated chair color");
      });
    });
  };

  return (
    <div className="App">
      <button onClick={changeBackgroundColor}>Change background color</button>
      <button onClick={changeChairColor}>Change chair color</button>
      <Viewer apiRef={apiRef} />
    </div>
  );
}

A new color for our chair!

Hooks refactor

To abstract the initialization code and clean up our component, we'll write a custom hook that leverages all the code previously written plus a useState for the api object.

Custom hooks are one of the most powerful tools to keep our code well organized. Once you start using them, there's no going back.

import React, { useEffect, useRef, useState } from "react";

// Our wonderful chair model
const MODEL_UID = "c632823b6c204797bd9b95dbd9f53a06";

const useSketchfabViewer = () => {
  // This ref will contain the actual iframe object
  const viewerIframeRef = useRef(null);
  const [api, setApi] = useState();

  const ViewerIframe = (
    <iframe
      // We feed the ref to the iframe component to get the underlying DOM object
      ref={viewerIframeRef}
      title="sketchfab-viewer"
      style={{ height: 400, width: 600 }}
    />
  );

  useEffect(
    () => {
      // Initialize the viewer
      let client = new window.Sketchfab(viewerIframeRef.current);
      client.init(MODEL_UID, {
        success: setApi,
        error: () => {
          console.log("Viewer error");
        },
      });
    },
    // We only want to initialize the viewer on first load
    // so we don't add any dependencies to useEffect
    []
  );

  return [ViewerIframe, api];
};

export const Viewer = ({ apiRef }) => {
  const [ViewerIframe, api] = useSketchfabViewer();

  apiRef.current = api;

  return ViewerIframe;
};

What we learned

We can use refs to feed DOM elements to libraries that need them.
We can use refs to give back objects to parents. Another solution could have been to use a React Context.
We can use custom hooks to make our code cleaner and more readable.

You can find the final source code here.

Shout-out to Sketchfab for their amazing product and in particular to Arthur Jamain for the great collaboration we had over the last few months.

Can I Use Web Components

Fri, 10 Apr 2020 00:00:00 GMT

The goal of this article is to show you what is a web component, why you can now use it in production, and when you should do so.

For all those who already have some knowledge in web components, I would like to specify that I will be talking about the web component V1 standards as the v0 draft APIs are now removed from Chrome since Chrome 80.

If you’re new to web components, don't worry you can catch up starting from the v1 standards, looking at the v0 specifications and standards might not be useful unless you want to understand the journey of the web components to reach its V1.

But first, you might want to be able to understand the basic concepts of a web component.

Definition of a web component

So first and foremost, what is a web component?

"Web Components is a suite of different technologies allowing you to create reusable custom elements — with their functionality encapsulated away from the rest of your code — and utilize them in your web apps." - MDN

Simply put, the web component is not just a thing by itself. For a component to be called so, different technologies have to be combined.

What are those "different technologies" then?

I will go through the different technologies that make a web component. This article won't be able to encompass all the possibilities of a web component and all its possible implementations. For those who would like to go further into the explanations, I will provide some resources as you read through.

Custom elements

A web component is nothing more than an enhanced custom element. A custom element is a javascript class expanding the HTMLElement class that allows you to create a new HTML tag for your component.

First, you have to create a class as shown in the script below.

class MyComponent extends HTMLElement {
  constructor() {
      super();
      ...
    }
}

We then define a custom element with the previously created class.

customElements.define("my-component", MyComponent);

Your custom element is ready to be used as a tag in any web project.

<my-component></my-component>

let's try it in a simple html document:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>my first custom element</title>
  </head>
  <body>
    <my-component></my-component>
    <script>
      class MyComponent extends HTMLElement {
        constructor() {
          super();
        }
      }
      customElements.define("my-component", MyComponent);
    </script>
  </body>
</html>

And that's it! you have created a custom element. Now if you inspect your HTML document on a web browser you should be able to see something like this:

The main idea of custom elements is that you can create, extend, and reuse them everywhere in your code.

If you are looking for a more in-depth documentation, check the official documentation. I would also recommend this article on customElements that is giving you a good overview of the matter.

Shadow DOM

The next technology that we will be discussing is the Shadow DOM. When you attach a shadow DOM to your custom element you then scope all the CSS, and the HTML in this DOM. The big idea of the shadow DOM is to be able to create templates for your web components that won't be affected by any style or js selectors and therefore keep your code and feature as intended.

Let's try it in our previously created component :

First create a shadow root:

const shadowRoot = this.attachShadow({ mode: "open" });

If you want to know more about the attachShadow mode argument I strongly recommend this article from Leon Revill.

Create some elements:

const spanElement = document.createElement("span");
spanElement.textContent = "Hello world";

Create a style element:

const style = document.createElement("style");

Create a css class for your span element and add some css rules:

style.textContent = ".my-class {color: red}";
spanElement.setAttribute("class", "my-class");

And finally attach them to your shadow root:

shadowRoot.appendChild(style);
shadowRoot.appendChild(spanElement);

so here it what your component should look like now:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>my first custom element</title>
  </head>
  <body>
    <my-component></my-component>
    <script>
      class MyComponent extends HTMLElement {
        constructor() {
          super();
          const shadowRoot = this.attachShadow({ mode: "open" });

          const spanElement = document.createElement("span");
          spanElement.textContent = "Hello world";
          spanElement.setAttribute("class", "my-class");

          const style = document.createElement("style");
          style.textContent = ".my-class {color: red;}";

          shadowRoot.appendChild(style);
          shadowRoot.appendChild(spanElement);
        }
      }
      customElements.define("my-component", MyComponent);
    </script>
  </body>
</html>

Now you can try going to the console and access and modify the elements in the shadow DOM but you won't be able to.

As well if you add some style in your HTML file, it won't affect the shadow DOM elements. Try adding style on the span element for example:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>my first custom element</title>
    <style>
      span {
        color: blue;
      }
    </style>
  </head>
  <body>
    <span>Hey! I am just here for the example tho...</span>
    <my-component></my-component>
    <script>
      {...}
    </script>
  </body>
</html>

As you can see the span inside the shadow DOM is not affected by the style.

HTML Template

"The template element is used to declare fragments of HTML that can be cloned and inserted in the document by script. In a rendering, the template element represents nothing. The template contents of a template element are not children of the element itself." - html.spec.whatwg.org

This means that you can define a template for your Web component and clone it for every instance of your component.

So first you create your template:

const template = document.createElement("template");
template.innerHTML = `
  <style>
    span { color: red; }
  </style>
  <span>Hey There! I am in the template </p>
`;

Then you can clone it's content into your shadow tree:

shadowRoot.appendChild(template.content.cloneNode(true));

As I am sure you noticed, we don't just add the template in the shadowRoot but instead we copy its content. That's because Templates are not rendered so what we need for our component is the content of the template.

Let's implement it in our component:

export class MyComponent extends HTMLElement {
  constructor() {
    super();
    const shadowRoot = this.attachShadow({ mode: "open" });

    const template = document.createElement("template");
    template.innerHTML = `
    <style>
        span { color: red; }
    </style>
    <span>Hey There! I am in the template </span>`;
    shadowRoot.appendChild(template.content.cloneNode(true));
  }
}

The important point to keep in mind about templates is that it allows you to declare chunks of HTML in your document that will be rendered and that sole purpose is to be cloned and reused. It is a perfect fit for web components.

You could define it in your index.html since template elements are not rendered in the browser but I would recommend keeping it close to your component declaration to keep your index.html file clean and organize your templates by component.

You can even use the slot Element to make your template more flexible. Those of you who already worked with Vue.js may be familiar with the concept of slot. For the others Slots are like placeholder. They allow the developper to create specific elements that will be placed inside the parents component DOM at predifined emplacement.

Imagine that your component template is a text with holes, each hole has an id. When you add children to your component, you can assign them a slot attribute. This slot attribute allows your component to place them in the "holes " according to their ids (the slot attributes).

If you want to learn more about slot element with web component I would recommend this documentation from MDN.

The addition of those four standards-based technologies is what defines the web component today.

ES Modules

This part is not always explicit in the definition of a web component. It doesn't affect the component on itself but more the way you will be able to use it in a project.

"The ES Modules specification defines the inclusion and reuse of JS documents in a standards-based, modular, performant way." - webcomponents.org

The ES modules should allow developers to use the script tag with the type module. Wich means you can export variables or components from your .mjs files and import it in your .html file within a module script tag.

The ES modules standard allow developers to reuse web component in any HTML file by importing them from an mjs file.

// wc.mjs
export class MyComponent extends HTMLElement {
  constructor() {
    super();
    const shadowRoot = this.attachShadow({ mode: "open" });

    const spanElement = document.createElement("span");
    spanElement.textContent = "Hello world";
    spanElement.setAttribute("class", "my-class");

    const style = document.createElement("style");
    style.textContent = ".my-class {color: red;}";

    shadowRoot.appendChild(style);
    shadowRoot.appendChild(spanElement);
  }
}

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>my first custom element</title>
    <style>
      span {
        color: blue;
      }
    </style>
  </head>
  <body>
    <span>Hey! i am just here for the example tho...</span>
    <my-component></my-component>
    <script type="module">
      import { MyComponent } from "./wc.mjs";
      customElements.define("my-component", MyComponent);
    </script>
  </body>
</html>

An MJS file is a source code file containing an ES Module.

You will need a server to be able to fetch with import, as it doesn’t work on the file:// protocol. You can use npx serve to start up a server in the current directory for testing locally.

If you are interested in how the ES module works I strongly advise that you read this article from Lin Clark.

Can I use it?

Now, a lot of developers are still quite reluctant to use web components in their projects. And you can understand why! The lack of common standards implemented by all major browsers since it's creation in the early 2010s has allowed many web component libraries to raise and grow sub-communities that do not share the same opinion on the definition of a web component and its implementation.

On the article Why the React community is missing the point about Web Components by Ben Halpern in November 2018, Dan Abramov comments the following :

[...] The problem is that there's no single "web component community". There are multiple sub-communities. You mention "web component libraries" in the post. These libraries don't agree on a single standard so React would need to pick a side in debates like "how does server rendering work with web components". Whatever we pick will have large downstream effects, and our reluctance to support more has to do with being careful [...] about the semantics — not somehow being at odds with web components per se.

So I think some important points have to be clarified for you to understand the implication of using it in your projects.

Web components are tools

It is important to understand that Web components are not designed to be better than HTML elements. We can consider web components as a tool to extend HTML, not to replace it.

And as we know the same tool cannot be used for all projects. So before adding this tool to your toolbox make sure you really need it or else it might some unecessary weight.

Here are some points that you should consider before jumping in :

It is not that simple to write a web component. It can be quite complex and verbose depending on the scope of your web components. And most of the project with quite a large number of web components have been using some libraries to creates them. webcomponents.org has a page dedicated to those libraries.
Comparing a web component and a react component for example is like comparing apples and oranges. The definition of a web component is not setting limits to what you can put in it. React in its documentation explain it this way:

"React and Web Components are built to solve different problems. Web Components provide strong encapsulation for reusable components, while React provides a declarative library that keeps the DOM in sync with your data. The two goals are complementary. As a developer, you are free to use React in your Web Components or to use Web Components in React, or both."

Understanding and defining what will be the interfaces between web components and the rest of your project's tools is important.

Web component v1 standards are not that old and you probably want your project to work on older browsers versions too. Simply put, you might need polyfills. I will come back to it later on.
One of the down points of the web component often cited is that developers have been struggling to use it with server-side rendering. This is mainly caused by the fact that the contents of ShadowDOM are often missing when the view is delivered to the client. One possibility is to remove the shadow dom from your component like you can see in this article. Other web component advocates found ways to make it work like in this article by Steve Belovarich or this one by Peter Goes. I think you can sum it up by saying that for now there are no standards for server-side rendering with a web component but you can always find a workaround.

Just remember to take all this point in account before jumping in.

Future proof

One of the main arguments of the pro web component community is that web components are future proof. Since it is based on web Standards, it is now supported by all modern browsers and should always be.

I can already hear your ask:

"Alright that's for the future? But what about our beloved past heroes like IE?"

Well you are right to be worried about them and that should be one of your main concerns.

Below are the previously cited technologies browser supports at the current date. If you want the latest data, click on the link below each of them.

can i use custom element v1

can i use custom shadow DOM v1

can i use javascript module script tag

can i use javascript module dynamic import

can i use HTML templates

As you can see neither Internet explorer is not supporting any of them neither the early versions of other browsers.

There are polyfills for the web component APIs. You can use the official polyfills.

With those polyfills, nothing you can now develop your future and past proof web component!

Reusability

A web component is easily reusable and is trustworthy. Its strong encapsulation provides security for its data. It also ensures that it will be used the way its designer intended to.

Should I use it?

Based on previous points, there are no specific reasons to not go for it. Still, the Web component isn't mature enough to be trusted for production by most developers. It solves problems but adds others.

For now, the best use the community can agree on for a WC-based project is a design system UI components library. Tesla for example, has crossed the bridge and chosen to make its design system with web components.

Building a sustainable design system can be a real brain teaser for companies. Big companies often use a lot of different technologies and frameworks for all their products.

They reach a point where sustainability becomes a problem in terms of design and technology. They ask themselves those questions :

Do we have to create a UI component library for each framework that we use?
How can we be sure that we don’t switch framework in the next few years?
And If we make sure we don't, aren't we missing an opportunity?

A web component is not based on any framework owned by any company but on standards adopted by all major browsers. You don't have to worry about the hype effect on your chosen technology or framework. This is why web components fit if you decide to create a UI component Library or even a design system.

Of course, you can also use it in any static pages project but I wanted to highlight the one big use where you should consider it.

Conclusion

Now that the standards that define Web components are implemented in every modern browser, they are meant to be part of the future of the web.

As a web developer, it’s time to consider web components as a practical solution for your features and problems.

I would like to quote Serhii Kulykov(@webpadawan) on this article

"Now in 2019, it’s time to give a try to Custom Elements and Shadow DOM. They might not be “the next big thing” by themselves, but rather solid foundation the next big thing could be built upon." - Serhii Kulykov

Of course when the technology will be more mature, accepted and integrated into our projects we will witness the apparition of new ideas and maybe even new standards.

But for now, we are the ones who have to be bold enough to try on our own.

If you are interested in the subject, I recommend the following articles:

The journey of Web Components: wrong ways, lacking parts and promising paths - Serhii Kulykov

Why I don't use web components? - Rich Harris (svelte creator).

I recommend you read the article but most importantly to read its comment. Especially this gist that is a direct answer to the previously mentioned article

Why the React community is missing the point about Web Components - Ben Halpern.

Once again the interesting part is mostly in the comments with a discussion between Dan Abramov et Benny Powers

Optimizing for high availability in time of crisis

Thu, 09 Apr 2020 00:00:00 GMT

These past few weeks have been tough for many websites. Some of them find themselves with an unprecedented increase in traffic (e.g. video conference software, online grocery stores, social media) and others with a dangerous reduction of users (e.g. car insurance, travel agencies, etc).

Although both of those scenarios present severe risks for businesses, the second one often cannot be solved by technology alone. So the goal of this article is to help with the former and provide a set of quick actions to maintain a highly available website on AWS under unexpectedly high traffic.

Most of the time in this situation, getting our website back online as soon as possible will be our top priority, which doesn't leave much time for writing or changing code (and would we want to add untested code into the situation anyway?). It also means that we're prepared to spend more on our infrastructure to accommodate the increased traffic (which may well pay for itself if that helps generate more business). For each recommendation I'll try to give a sense of the price increase involved.

Some of the actions below are pretty generic (only assume a standard 2- or 3-tier web infrastructure) and some are more context dependent. We'll cover the generic actions first and then explain in which context the specific actions apply.

Generic actions

1. Increase instance size

Time: < 1h

Money: from ×2 to ×100 on EC2 bill (you choose)

Effectiveness: increased capacity for cpu- or network-bound websites

This is the simplest and quickest action, although it can be a little bit expensive depending on the new instance type we choose.

Keep in mind that increasing instance size does require replacing all the currently running ec2 instances (we can't increase the size of an existing instance), so this migration requires some planning. In particular, if our application is stateful, we risk losing data so our best option is probably to take snapshots of EBS volumes and recreate them for the new instances. In most cases however all that's needed is a redeployment of the app on the new instance(s).

To save on ec2 costs remember to use reserved instances when possible.

2. Use autoscaling

Time: 2h - 6h

Money: from ×0.1 to ×10 on EC2 bill depending on autoscaling rules

Effectiveness: lifts all cpu- or network-bound blockers for stateless apps

This is a very good option if our app is already stateless.

Autoscaling will automate the creation and termination of EC2 instances to match the fleet size to the current traffic. This will allow us to not over-provision instances to serve our users and save a lot of money. Remember that we can set a max number of instances in our fleet so we can stay in control of billing.

If our app isn't stateless already, we may want to consider focusing development efforts to make it so in the short term. Although it's likely to require substantial change to the code, it's also going to unlock big performance improvements through parallelization (and autoscaling) so it might be worth the trade-off. If however we decide that making our application stateless isn't for us, there still are still things we can do to unlock performance, especially option 6.

3. Deploy to multiple regions (active/active sites)

Time: 0.5 day (with IAC, prepared) to 3 days (without IAC, unprepared)

Money: ×2 for the entire AWS bill

Effectiveness: doubled capacity

Another way to keep response times low worldwide and to massively increase our website's resilience is to deploy it in multiple regions, both serving traffic (also called active/active deployment). By this point we should already be using multiple availability zones in the first region and, by deploying a copy of the site to a second region, we will get:

the capacity to serve double the normal number of users ;
a site that's resilient against region-wide disasters.

Infrastructure as code (or IAC, with e.g. Terraform or CloudFormation) will make this deployment much easier and faster. If we don't use such a solution already, I wouldn't recommend setting one up just now, as it usually takes longer to prepare an IAC script than to provision an environment manually through the console. However we should definitely add this to our backlog for when things start to calm down.

Once we've got our second region running, we'll need to split the traffic between the sites. This can be done with Route53 which has several routing policy types to handle this, including latency-based or geolocation routing.

Finally we probably need to use a "single" database for both regions. One way to achieve this is to create a read-replica of the existing RDS database in the second region (with async replication). Point all database access to the master database in the first region and in case of disaster in zone 1, we can promote the replica (AWS replaces the dns automatically, no code or configuration changes required).

Specific actions

4. Cache static assets

Time: 0.5 to 1 day

Money: depends on asset sizes and volume downloaded, see calculator

Effectiveness: removes static files load on web servers

This is useful if web servers take time to serve each and every request for static assets (e.g. images, CSS and JS scripts, static html) or, even worse, re-generate them each time. In that case a great way to reduce the load on the servers is to have AWS serve the static assets for us.

A standard solution to this problem is to store assets in S3 and serve them through AWS's CDN CloudFront. Using CloudFront enables caching of resources and uses edge locations around the world to ensure fast response everywhere, however it becomes expensive with many users. Use AWS's simple monthly calculator (the newer AWS pricing calculator doesn't support CloudFront yet) to estimate how much it would cost.

5. Cache long-running database queries and computed value

Time: 1h to 1 day depending on web framework

Money: 15 USD/month (with instance reservation) to 20k+ USD/month depending on requirements

Effectiveness: removes expensive computations from web servers

It's an almost unavoidable issue on dynamic websites with growing data: unoptimized database queries start taking longer, eventually blocking worker processes for several seconds at a time and seriously limit our ability to scale the number of concurrent users. While it's important to optimize database queries as much as possible, there's a simple way around this issue: caching query results.

A typical way to implement application-level caching on AWS is ElastiCache. We'll pick between a Redis and a Memcached backend based on what's easiest to implement in our application code and use the cached value as often as is reasonable.

The same reasoning (and solution) applies for computed value that are not user-dependent and don't need to be recomputed on each request.

6. Store sessions data properly

Time: 1h to 1 day depending on web framework

Money: 10 USD/month (with instance reservation) to 1k+ USD/month depending on number of users

Effectiveness: removes lock acquiring bottleneck

In case our current application has to remain stateless and we're storing session information on the servers, make sure to store the session data in a way that allows concurrent access to the session store.

For example, in some implementations (especially in PHP), storing sessions in a single file can lead workers processes to waste a long time acquiring the session file lock on every request. An alternative to that is to store session data in memory, for example using ElasticCache on AWS (see action 5).

7. Rewrite bottleneck services

Time: depends on service

Money: however much dev time worth of money

Effectiveness: removed application level bottleneck

This one is for last resort, when no other options are obviously available. Once we've identified (one of) the subsystems that are responsible for performance bottleneck(s) (for example using AWS X-Ray, New Relic or plain logs), it's time to consider whether a rewrite would yield significant improvements.

In this sort of situation it's beneficial to start relying on cloud services to be able to build quickly and cleanly. AWS services like SQS (for decoupling), AWS Chatbot, Elastic Transcoder, Lambda, Amazon Polly and Kinesis for example are reliable building blocks to build solid, serverless workflows for chatbots, video transcoding, natural language processing or event streaming quickly. And there are many similar services we can use to build extremely scalable applications.

Need more help? I'd like to help you with your scalability issues! Contact me by email or on twitter and we'll try to find a solution that's right for you. 🙂

How DUOLAB uses artificial intelligence to improve your skin

Fri, 03 Apr 2020 00:00:00 GMT

For a long time people (including myself) have used the same skin care routine everyday. Only occasionally do we switch up our products for a different brand. The creams are opened and resealed multiple times a week, resulting in the products containing preservatives and other unnatural ingredients to prevent them spoiling. Innovative new start-up Duolab aims to improve the way we do skincare, their slogan being

‘Your life is no routine. Why would your skin care be?’

In November 2019 I started working for Theodo UK, a London-based software development agency which delivers bespoke web and mobile solutions for both start-ups and world-renowned clients alike. I was very excited to hear that my first project would be working within a scrum team to deliver DUOLAB’s web and mobile application to accompany the launch of their cutting-edge skincare product. Having a strong interest in skincare myself, I was satisfied with the project I was about to embark on!

Everybody’s online presence and engagement is more important than ever given the current Coronavirus pandemic and home lockdown! DUOLAB is ahead of the curve by creating a future proof product to personalise your skin care without the need of a dermatologist or a shop assistant to provide a recommendation. You’re now able to improve your skin from the comfort of your own home.

DUOLAB’s artificial intelligence (AI)-driven personalised skin care is revolutionising the way we care for our skin. Unlike traditional skincare treatments, DUOLAB utilises an AI skin diagnostic tool within its mobile app to calculate a tailored, personalised skincare protocol.

The DUOLAB device blends a suitable combination of a moisturising base and a targeted concentrate which has been recommended to the customer by the skin diagnostic tool. The skin diagnostic process begins with a guided selfie which is inputted into the AI model. The AI assesses various skin traits, such as blemishes and wrinkles, which gives rise to more accurate recommendations than ever before. The selfie is followed by a short quiz which further determines the base-concentrate combination.

Your skin changes over time, so customers can retake the skin diagnostic when they choose - this will give them an up to date recommendation.

Theodo UK’s comprehensive technology stack allows us to approach a broad range of clients with solutions to their business needs and problems. In the development of the DUOLAB mobile app, we used Facebook’s React Native framework to create a cross-platform, native app with an excellent user experience. Our use of Amazon web services (AWS) for hosting the mobile app and website was crucial in getting the project set up and running quickly. AWS gave us the reliability and flexibility that we needed to build our products with confidence.

Our best-in-class products and quick turnaround time are testament to our position in the global start-up studio, M33. Theodo’s M33 association enables us to leverage our relationships with other M33 companies to diversify our final product offerings. During the DUOLAB project, we worked alongside fellow M33 companies, Sicara and BAM Mobile Limited, who were responsible for the skin AI analysis and mobile app designs respectively. The shared philosophies of the seven M33 companies has enabled them to develop a robust, synergistic relationship which is capable of tackling the digital needs of modern companies.

In January 2020, DUOLAB introduced their skincare solution to the world at CES® 2020 in Las Vegas - ‘The global stage for innovation’. The mobile app was demonstrated to consumers before its launch and it was well received.

Since then DUOLAB has launched the product, mobile app and website within the UK. The products are on sale in selected L’Occitane stores and on the DUOLAB website. Given the current situation, DUOLAB has adapted to the needs of the consumer in a time when many other competitors haven’t. I hope that you can take these lessons from DUOLAB and stay ahead of the curve.

Serverless: a New Paradigm

Fri, 03 Apr 2020 00:00:00 GMT

Since its creation in 2014, AWS' Serverless has delivered by reducing the cost of infrastructure management while increasing speed, agility, and scalability.

What is Serverless?

“Serverless allows you to build and run applications and services without thinking about servers” — AWS

Serverless shifts more responsibility to Cloud Providers, allowing IT teams to build applications and services without managing the underlying architecture. Developers can send code, run databases, store data and manage security, without thinking in terms of servers, networks and patching. One classic example of a Serverless service is FaaS (Function as a Service), a Serverless approach to Compute that allows developers to send application code to their Cloud Provider and have it run without dealing with the underlying complexities of the server or network or other infrastructure components.

Today the three main Cloud Providers for Serverless services are Amazon Web Services, Microsoft Azure, and Google Cloud.

“A Serverless solution is one that costs you nothing to run if nobody is using it.” — Paul Johnston, ServerlessDays CoFounder

Prerequisites for Choosing Serverless in Your Organization

Splitting Your Business Logic Into Events

The first step in architecting an event-driven system is to understand the business domain through its events. Domain-Driven-Design (DDD) is a great framework to achieve this and Event Storming workshops can help business and technical stakeholders understand the processes and systems at play in a given domain.

These events are represented in business terms, e.g. Order Placed. But they ultimately trigger different Serverless services, for instance, running code, writing data, and authentication.

Code Compatibility

Because the feature is hosted and deployed on a Serverless Cloud Provider, it must follow its rules.

As you shift more responsibility to your Cloud Provider, there are some constraints around the code you write. Primarily, the language teams use has to be compatible with their chosen Cloud Provider. For example, Google Cloud only accepts functions written in Go, Node.js, and Python. However, custom runtimes make any language possible.

Serverless Pros and Cons

Pros:

Automatic scaling

The Cloud Provider ensures that the function responds consistently every time it is called. For example, iRobot faces a huge peak at Christmas when new robot owners start their device, AWS will automatically scale compute power to absorb the load.

In a Serverless paradigm, the Cloud Provider manages the running of your services, eliminating manual provisioning.

You only pay for what you use

Serverless Cloud Providers charge based on resource usage only as opposed to paying for allocated servers that most of the time are only used at a fraction of their capacity. No need to pay constantly for the extra buffer of compute power you might need during peak periods. And even better, if your function isn't used, you don't pay.

When you consider that organizations use only a portion of the actual computational power of their servers, you realize the potential gain is significant. However, these servers that sit unused still consume energy, which leaves room for improvement to make the cloud computing industry greener.

Reduce the Ops team size needed to run your systems, reducing your TCO (total cost of ownership).

As there is less infrastructure managed by the team, they can focus on delivering value to users, rather than running the underlying systems.

Source: Serverless.com

Cons:

Vendor Lock-In

As you become serverless, your systems couple to the underlying Cloud Provider. This makes it difficult to move to another Cloud Provider later on, for instance, if their pricing changes. Certain large organizations are concerned by this dependency and therefore want to remain cloud agnostic. There are some solutions to reducing the extent of vendor lock-in, but these often come at the cost of some original benefits of Serverless.

Requires a mindset shift (a new way of coding)

Serverless can be massively empowering to developers, but it requires a mindset shift, correct coding, and consistent approaches to security and compliance. In this video, Nicole Yip, Senior Infrastructure Engineer at Lego, explains how their journey to Serverless was tightly linked to instilling the DevOps culture within the team.

Changes certain security practices

Serverless removes certain security concerns, like patching operating systems as it is handled by the cloud provider who seamlessly applies security patches between two executions of a serverless function. Other security practices remain unchanged, like using a Web Application Firewall (WAF). But serverless also introduces an unprecedented level of flexibility in the definition of privileges at the function level. For instance you can specify that a given function should only be able to query certain elements of your database. This flexibility can prove itself dangerous as a rushed developer might be tempted to set lax security rules. This is why serverless requires adequate coaching and automation to ensure those rules are properly set.

Find out more on security with our expert Ben Ellerby’s talk at ServerlessDays.

Cautions

If your code is used at high intensity, the Serverless solution might cost more for each query than the marginal cost your organization pays for conventional hosting.

Parts of your application that would not benefit from permanently allocated resources can still be outsourced to a Serverless function. Authentication is a classic example where you would use a serverless authentication service like Cognito.

Use Cases

Serverless architectures have flourished. And use cases have grown in a broad range of industries and companies of all sizes such as:

Netflix: Uses AWS for automatically encoding media files, automation of backup management, monitoring of other AWS services.
PayPal: Has moved mission-critical workload to Google Cloud and transformed the way its 5,000 developers work.
CodePen: A single SRE (Site Reliability Engineer) manages a Serverless infrastructure capable of responding to more than 200,000 requests in times of high demand, reducing the cost of managing infrastructure.
Lego: After a system failure during Black Friday, the team decided to move to Serverless to handle peaks for their online shop. Their team now quietly watches their systems auto-scale.

In-House Serverless (An Alternative to Cloud Providers)

Since 2016, various frameworks have been gaining momentum, including OpenFaas and Knative. These solutions make it possible to no longer depend on Cloud Providers like AWS or Azure and allow you to replicate the code execution aspect of Serverless applications based on Kubernetes to host them yourself. However, your organization is again responsible for the servers on which your code is run.

Large corporations usually have large IT departments, maybe a few data centers and often an intricate plan to migrate to the cloud. Such companies could benefit from the speed and agility provided by Serverless Compute (Functions as a Service or FaaS) that would be enabled deploying Knative on their own infrastructure. The complexity of managing would not be outsourced to a Cloud Provider but handled by the existing IT team but the development team would highly benefit from the possibilities that Serverless has to offer.

Conclusion

Serverless is recognized for bringing more agility and scalability to the organization. In addition, Serverless can help large organizations reduce the time-to-market of new products.

By breaking up a monolithic architecture in microservices, developers can focus on business-specific code and deliver customer-centric solutions. Where appropriate, it is an excellent solution to reduce the cost of managing the infrastructure and increasing the impact of your developers on your business.

Whether you want to start a new project with state-of-the-art technology or are looking to reduce the cost of managing your infrastructure, Serverless is a technology to seriously consider.

Ensure code quality; create your own ESLint rules

Wed, 01 Apr 2020 00:00:00 GMT

Ensure code quality; create your own ESLint rules

Introduction

We used ESLint rules to enhance code quality. This article is about how one would proceed to create one's own custom rule. We used this procedure at Theodo to make sure our frontend code was XSS free on every project.

If you are not familiar with XSS vulnerabilities here is an article from OWASP that summarizes well what it is about.

Context and Needs

There are over 80 developers at Theodo, working on over 30 different projects. This setup presents some challenges that many tech companies face:

Code consistency and maintainability.
Continuous improvement of the code quality
Optimization of the development environment
A straightforward onboarding for the newcomers on the different projects

We came up with a solution that is easy to implement and extremely effective: creating a custom linting rule for ESLint.

I will use an example of a rule we needed for our frontend projects with an objective of no know XSS vulnerability in our code.

We started by implementing the rule to React and Vue, which covers the majority of our projects. There are a few differences to take into account when developing a rule for each of these frameworks. We will look at them in this article.

If you encounter issues configuring ESLint, especially alongside other fromatting tools such as Prettier, I recommend this article that tremendously helped me to configure my development environment.

Strategy

Nobody is infallible. Creating an automated check that alerts the developer is more reliable than documentations on the project or code reviews. Don't get me wrong, code reviews are essential, as is a well-written documentation, but it is often not enough to enforce a coding practice. Having an automated rule also means shorter code reviews and an easier time following the best practices.

Implementing a rule that will be in the IDE also allows you to suggest the best way to do something, as soon as it is written. This is a roundabout way to train your developers, that isn't time-consuming for you.

How to implement an ESLint rule

To understand how a rule is coded, we need to take a look at our code's Abstract Syntax Tree and how we can use it.

The Abstract Syntax Tree (AST)

An AST is a graph representation of your code. ESLint uses it to navigate your code and to decide which actions to take and where.

The screenshot below is an example of a simple Hello World! AST of a React script, containing a dangerouslySetInnerHtml. We can see the tree structure and the names of the different nodes. For example, the FunctionDeclaration node contains the information from the function keyword, up to the closing tag.

The screenshot below is an example of a simple Hello World! AST. We can see the tree structure and the different nodes' names. For example, the FunctionDeclaration node contains the information from the function keyword, up to the closing tag.

This screenshot is taken from astexplorer.net which is an extremely useful resource to visualize and navigate your AST.

It works nicely for Javascript and React. However, when exploring your Vue AST, you will need to find another way for the <template> part of a file, which is parsed as HTML. AST explorer's own HTML configuration does not match perfectly with what my Vue code outputs. I used this tool instead, which is the same, except only for Vue.js

We'll come back to the particularities of Vue.js later in this article.

How ESLint uses your code's AST

ESLint uses a visitor that will parse your AST for each file it is allowed in. In this visitor function, you can define actions for each node type. If we use the same example as above, we could write something like this.

    const create = context => {
      return {
        Program(node) {
          try {
            console.log('Entering the program!');
          } catch (error) {
            context.report(node, error.stack);
          }
        },
        FunctionDeclaration(node) {
          try {
    				console.log(`Function ${node.id.name} is defined`);
    			} catch (error) {
    				context.report(node, error.stack);
    			}
    		}
    	}

In the example, I use the root node Program to trigger some action, and the FunctionDeclaration node.

This is a very basic example, where we can already see a few important things about the way ESLint will work.

The context object

The context passed as an argument here contains information relevant to the rule. We won't use it here, but if you want more info here's the official documentation: https://eslint.org/docs/developer-guide/working-with-rules
ESLint uses the node names to trigger some actions. Here, a console.log. These actions will be triggered for each node with the same name in the AST. For example, there is only one Program, but usually more than one function declaration. In the above example, our console output will look like:
```
Entering the program!
Function example is defined
Function helloWorld is defined
```
Each node has access to its sub-nodes, and also to its parent node if it has one. This is why we can use node.id.name to get the name of the declared function. The Identifiernode 'id' is a sub-node of the FunctionDeclarationnode, just like the BlockStatement node named 'body'.
Our approach to implementing our custom rule

Test-Driven Development works like a charm in this particular case. We obviously want to test our rule, but there are other advantages to doing so.

For those who are not familiar with Test Driven Development (TDD), it consists of three phases.

Write a test case you want your code to pass. Run the tests. It fails. (Red test)
Write code to pass your test, not more. Run the tests, it should pass. (Green test)
Refactor your code.

This process is often shorthanded in Red-Green-Refactor.

The TDD method is awesome and you should use it where it fits. It does fit here for a few reasons.

One of them is that it allows us to ship a stable and well-documented version, with examples based on the tests.
In the context of our project, we were able to quickly test our custom ESLint rules in real-life projects, and obtain feedback. It was easier to convert the feedback in tests first (red tests), and then write some code to fix the issue (green tests).
In our case, many of our tests were easy to write, since they often were chunks of code we wanted to be flagged, or not. However, turning them green is less trivial. Here's an example:

    // Test cases we do not want to catch with our rule
    testCase(`
          <template>
            <div class="content">
              <div v-html="message" />
            </div>
          </template>
    
          <script>
            import DOMPurify from 'dompurify';
            const rawHtmlInput = '<a onmouseover=\"alert(document.cookie)\">Hover me!</a>';
            export default {
              name: 'HelloWorld',
              data () {
                return {
                  message: DOMPurify.sanitize(rawHtmlInput)
                }
              }
            }
          </script>
        `),
    
    // Test cases we do want to catch
    testCase(`
          <template>
            <div class="content">
              <div v-html="message" />
            </div>
          </template>
    
          <script>
            import DOMPurify from 'dompurify';
            const rawHtmlInput = '<a onmouseover=\"alert(document.cookie)\">Hover me!</a>';
            export default {
              name: 'HelloWorld',
              data () {
                return {
                  message: rawHtmlInput
                }
              }
            }
          </script>
        `),

These two cases are similar, except in the message part, one is sanitized by DOMPurify whereas the other isn't.

And more generally, tests protect us from breaking previously working code, and catch some edge cases during the development phase.

Managing the Vue AST

As teased above, let's look at the particularities of Vue files when writing an ESLint rule.

Vue files are split in 3 parts, a <template>part, a <script>part and a <style>.

The AST, in this case, does not include a unified version of the file. It is split in the same way. This means our visitor function will have to be split as well.

I use a standard ESLint parser for Vue, that exposes a function which takes the context and 2 visitors as arguments. One for the template, and one for the script.

Here is a code snippet to give you a better idea:

    const create = context => {
      // The script visitor is called first. Then the template visitor
      return utils.defineTemplateBodyVisitor(
        context,
        // Event handlers for <template>
        {
          VAttribute(node) {
            try {
              // SOME CODE HERE
            } catch (error) {
              context.report(node, `${utils.ERROR_MESSAGE} \n ${error.stack}`);
            }
          },
        },
        // Event handlers for <script> or scripts
        {
          Program(node) {
            try {
              // SOME OTHER CODE THERE
            } catch (error) {
              context.report(node, `${utils.ERROR_MESSAGE} \n ${error.stack}`);
            }
    			}
        },
      );
    };

From there it is basically the same approach as a single visitor function, like with vanilla Javascript, duplicated in each function.

I used the library eslint-vuejs (https://eslint.vuejs.org/developer-guide/#working-with-rules) that already provides a function to manage the template and script part at the same time :

context.parserServices.defineTemplateBodyVisitor(visitor, scriptVisitor) where both visitor and scriptVisitor are functions like the one presented before.

And there you go, you can check any number of things you want to alert your user about.

In our Vue example, we checked for v-html in the template and checked if it was verified with libraries like DOMPurify. We know v-html is generally not the way to go, but sometimes it is a necessity, and we wanted to make sure we weren't introducing a potential XSS.

Here is what the custom rule looks like in action:

If you also think that integrating a custom ESLint rule to produce better code is awesome, definitely check out the project I used as an example, it open-sourced here: https://github.com/theodo/RisXSS

Conclusion

In conclusion, integrating custom ESLint rules in your projects will help improve the code quality, the lead time of your features, since fewer mistakes will make it to the code review part of the process, and the developer's experience.

With a rule matching a bad practice, you can easily suggest a good practice right in the IDE, and the developers will quickly and painlessly learn it.

In our case, after some pilots, the rule was used in the development environment of the developers and guarantees us a certain level of code security.

If you want to dive deeper into the custom rules world of ESLint, a good place to start is the official documentation for developers: https://eslint.org/docs/developer-guide/working-with-rules

I will also link here two articles that helped me greatly with ESLint custom rules. If you found mine interesting, you should definitely check these out.

Writing Custom Lint Rules for Your Picky Developers

Custom ESLint Rules For Faster Refactoring

How real developers use a CMS to build a showcase website with Netlify, Nuxtjs, and Contentful

Tue, 24 Mar 2020 00:00:00 GMT

Why build a CMS driven showcase website?

Here you are, you just started a project to build the latest shiny showcase website for your client. Now comes the difficult part: you want to offer your client the best bang for his bucks, not wanting to redo part of the website each time the content changes. In the same way, you are probably not in charge of coming up with the shiny design but coding it, why would you lose time adjusting wordings, coming up with translations?

That's where a Content Management system (CMS) becomes handy, giving your client an interface to edit the content of his site without wasting its precious developers' time. Now you might be thinking 'A CMS, last time I gave a go-to WordPress, I wasn't convinced !' and I agree with you. Today CMS fall into 2 usages:

A what you see is what you get, that generates a website: WordPress, Drupal are some examples
headless CMS which are a back-end only content management system built as a content repository that makes content accessible via a RESTful API: WordPress, Drupal are part of the game here as well, but we will use in this tutorial a concurrent specialized for this usage: Contentful to convince you that working with a CMS can be enjoyable.

Why use a server-rendered static website for a showcase website?

Doing so leverages the advantages of using a progressive web application with server-side rendering (more about SSR here) without the cost of running a server since you just need to serve your files. Furthermore, we leverage the following advantages for a showcase website:

SEO friendly
Performance
Monthly infrastructure cost

Why use Netlify & Nuxtjs

Netlify

Infrastructure wise our requirements are very small, we only need a CDN to distribute our site. We could use any major cloud provider directly, however, I decided to go with Netlify service for the following reasons:

The integration is exceptional: creating a CI that builds and deploys your website in production has never been easier.
Https works out of the box as long as you have your own domain.
Netlify service is built on top of AWS which means we don't have to worry about reliability.

Nuxtjs

Nuxtjs is a server-side rendering framework built on top of Vuejs. For our case, Nuxtjs allows for dead easy creation of a server-side rendered static website.

Building a CMS driven showcase website

Creating a Contentful CMS

First, as the website is content-driven, we will start by creating the CMS using Contentful to provide content to the website.

Start by creating an account on https://www.contentful.com/sign-up/ (use free trial for this tutorial) Once you sign up, you can create a new space (which is Contentful way of creating a project)

In the popup dialog set the type of space (you can upgrade later), the space name (the project name):

Space type: free
Space details: data_driven_static, empty space

You have successfully created a new contentful project

Creating the first content

Contentful is organized between:

Content Models which is where you define the structure of the content
Content which is where you can contribute (create content)
Settings where you will find some useful settings later on.

Create the content model

To create our first content we will use the Content model tab, click on add content type

When creating a new content type, we have to fill in some important information:

The Name of the content type, which is the displayed name for the content type: we will use Section.
The API Identifier is the name of the entity when querying the Contentful API. In most cases it’s better to keep the given API Identifier, to stick as close as possible to the content type name to ease identifying the content types in the code.
Lastly, I recommend filling in the description of the content type, I will use: Basic block of content for a page.

Finally, confirm to create the new content type.

Now, we have a new empty content type. To fill it, we will add some fields:

A Title for the section
A Text for a paragraph

To do so, first, click on add field:

For the type of the field, select: Text
For the settings of the Text field:
- Name: Title
- Short text
Click create and configure
Then in settings:
- Select: This field represents the Entry title
Switch to validation:
- Make it required
Finally switch to the Appearance tab:
- Leave it as a single line
- Enter a Help text: The section title
Click Save

You successfully created the first field, we will create a second one for the paragraph:

Click add field
In creating:
- Select type Text
- Name: Paragraph
- Long text
In Appearance
- Select Multiple lines
Click create and configure
Click Save

Once you created both fields, you can now hit save on the top right corner of the content type as we finished creating our content type.

Contribute the first section

As we now have the first content type, we can contribute to creating new content. To do so, in the main contentful header, head to the tab Content.

By default contentful filter with the current content type, we can create a new section hitting the button Add Section

You end up on the form to edit the entity. You need to provide a title and you can provide some text for the paragraph. Then hit publish.

You are done for the CMS part (keep it open for latter).

Installing Nuxtjs

First, we are going to create a Nuxt application.

To do so, you need to have Npm installed and type in:

npx create-nuxt-app <project-name>.

Once you launch the command, you will need to answer a few questions:

Project name: cmsdrivenstatic
Project description: /
Author: /
Package manager: / (I will use Npm)
UI framework: you can use any of them as you like (I will not use one here)
Custom Server framework: None
Nuxt.js modules: Axios, Dotenv
Linting tool: as you like, I go with and recommend all of them (ESLint, Prettier, Lint staged files, Stylelint needs some configuration hence I will not use it here)
Test framework: I will not talk about tests here, but it’s obviously recommended to use tests (both Jest and AVA are great)
Rendering mode: Universal (SSR)
Dev tools: use jsconfig.json, if using vscode as your IDE

You can run the Nuxt dev server using:

cd cms_driven_static
npm run dev

You are done installing Nuxtjs

Fetch the section content in Nuxtjs

We will now display the Section content in the Nuxt frontend. To do so, we need to install the contentful module: npm install contentful --save.

Then to use the contentful package in Nuxt, we will create a plugin. Plugins are the recommended way to use javascript libraries globally.

Finally, we will use this plugin to fetch content in the application pages.

Creating the Nuxt.js plugin

In the plugins directory of the project create a new file: contentful.js

Add the following code in the file:

// First we import the contentful node module
const contentful = require('contentful')

// Those are set via `env` property in nuxt.config.js or environment variables
const config = {
  space: process.env.NUXT_ENV_CONTENTFUL_SPACE,
  accessToken: process.env.NUXT_ENV_CONTENTFUL_ACCESS_TOKEN
}

// Create a client to setup fetching content
const client = contentful.createClient(config)

// Our first method to fetch all section content type
client.getSectionContent = () =>
  client.getEntries('', {
    content_type: 'section'
  })

export default ({ app }) => {
  // Add the function directly to the context.app object
  app.contentfulClient = client
}

Now we will register the plugin in our Nuxt config, head to the file nuxt.config.js. In this file you will find an empty plugins array. Add { src: '~/plugins/contentful' }, in this array:

-  plugins: [],
+  plugins: [{ src: '~/plugins/contentful' }],

That's all for the plugin configuration.

Finally, we need to specify the contentful credentials, to do so, we will use the dotenv module:

In nuxt.config.js add '@nuxtjs/dotenv' in the buildModules array:

  buildModules: [
    '@nuxtjs/eslint-module',
+   '@nuxtjs/dotenv'
  ],

Then in your .env file, you need to add two variables: NUXT_ENV_CONTENTFUL_SPACE and NUXT_ENV_CONTENTFUL_ACCESS_TOKEN:

Using the prefix NUXT_ENV_ enables Nuxtjs automatic injection of environment variables into process.env.

To fill in those values go back to contentful. Go to the settings tab and find ‘api keys’. If one already exists, use it, else create a new one. (specify any name and description)

Use both Space ID and Content Delivery API - access token to fill in your .env variables.

Restart your server (interrupt and npm run dev), the webpage should load errors free.

You should be set to fetch your content on your Nuxtjs page.

Display the section content

Let’s make this API call and display the section content.

Head to the file pages/index.vue. It’s the entry-point of your nuxtjs application, namely the landing page.

A brief introduction to Vuejs single file component syntax

A single file Vue.js component is organized in 3 parts:

<template>
  <p class="paragraph">{{ greetings }} World!</p>
</template>

<script>
  module.exports = {
    data() {
      return {
        greetings: 'Hello'
      }
    }
  }
</script>

<style scoped>
  .paragraph {
    font-size: 2em;
    text-align: center;
  }
</style>

The template part which is the rendered HTML for the component
The script part which contains the component logic
The style part which is, no surprise, the CSS style applied to the component.

Fetching the section content

First, we are going to fetch the section data and we want to fetch it server-side to be able to render the full HTML before sending it to the client.

To do so add an asyncData method below the component in the script HTML element, ending with the export looking like this:

export default {
  components: {
    Logo
  },
  async asyncData(context) {
    const sections = await context.app.contentfulClient.getSectionContent()
    console.log(sections.items)

    return {}
  }
}

Reload your page http://localhost:3000/, you should see your Section item in the console.

Then we will add some logic to the contentful plugin to map the interesting properties for the landing page Vuejs component:

// First we import the contentful node module
const contentful = require('contentful')

// Those are set via `env` property in nuxt.config.js or environment variables
const config = {
  space: process.env.NUXT_ENV_CONTENTFUL_SPACE,
  accessToken: process.env.NUXT_ENV_CONTENTFUL_ACCESS_TOKEN
}

// Create a client to setup fetching content
const client = contentful.createClient(config)

// Our first method to fetch all section content type
client.getSectionContent = () =>
  client
    .getEntries('', {
      content_type: 'section'
    }) // map the interesting properties from section
    .then(({ items }) => {
      return items.map((section) => {
        return {
          title: section.fields.title,
          paragraph: section.fields.paragraph
        }
      })
    })

export default ({ app }) => {
  // Add the function directly to the context.app object
  app.contentfulClient = client
}

And use it in the asyncData of pages/index.vue:

 async asyncData(context) {
    const sections = await context.app.contentfulClient.getSectionContent()
    console.log(sections)

    return {}
  }

Display the section

Finally let’s display our data, first make the sections available in the state of the component:

async asyncData(context) {
  return {
    sections: await context.app.contentfulClient.getSectionContent(),
    }
}

Update the template to use the new data from the asyncData:

<template>
  <div class="container">
    <div class="logo-container">
      <logo />
    </div>
    <div v-for="section in sections" :key="section.title" class="section">
      <h1 class="title">
        {{ section.title }}
      </h1>
      <h2 class="subtitle">
        {{ section.paragraph }}
      </h2>
    </div>
  </div>
</template>

Update the style of the component:

<style scoped>
  .container {
    margin: 0 auto;
    min-height: 100vh;
    display: flex;
    flex-direction: column;
    justify-content: center;
    align-items: center;
    text-align: center;
  }

  .logo-container {
    height: 50%;
    width: auto;
  }

  .section {
    margin-top: 20px;
  }

  .title {
    font-family: 'Quicksand', 'Source Sans Pro', -apple-system, BlinkMacSystemFont,
      'Segoe UI', Roboto, 'Helvetica Neue', Arial, sans-serif;
    display: block;
    font-weight: 300;
    font-size: 60px;
    color: #35495e;
    letter-spacing: 1px;
  }

  .subtitle {
    font-weight: 300;
    font-size: 21px;
    color: #526488;
    word-spacing: 5px;
    padding-bottom: 15px;
  }
</style>

Then reload your homepage, you will be able to see your new section content:

Congratulations

Let’s add one more section content in Contentful to check that our page works correctly.

Once you add the new section content and publish it you should see both sections on your homepage:

Deploy the website using Netlify

Setting-up automatic deployment

Finally, we will deploy the website, making it available on the Internet.

First of all, create a repository for the project on Github or Gitlab and push your project.

Once you are there, you can head to Netlify website: https://www.netlify.com/, create an account using your favorite authentication: https://app.netlify.com/signup

Once on the project page, I recommend using ‘New site from Git’ to setup up a static website deployment with the added benefit of an out of the box working CI, building your site with every new commit on your production branch.

Select your repository: If it doesn't appear in the list click on Configure the Netlify app on Github

For your site settings, go for:

Base directory: not set -> useful if you use a mono repository to set the directory where you can build the front
Build command: npm run generate
Public directory: dist
Click show advanced to add environment variables matching the ones defined in .env:
- NUXT_ENV_CONTENTFUL_SPACE
- NUXT_ENV_CONTENTFUL_ACCESS_TOKEN

Your site is now online!

You can find the Netlify generated URL (ending in .netlify.com) on the Netlify project overview.

Using your own domain name

Here again, Netlify eases the process, you simply need to have a domain name registered.

Once you have a domain name, you can carry on the settings:

Click on Set up a custom domain
In custom domain, click Add a domain alias and fill in the input with your domain name
Then you will need to point your domain to netlify servers, to do so, use the provided CNAME record in your DNS provider interface
After a few minutes to hours depending on your DNS provider, you should be able to access the website using your domain name.

Using TLS (HTTPS)

Once you can access the website using your domain, you will want to use https for security, SEO and performance purposes, you can and should use netlify https functionality:

Head to https in domain settings of your website:

Then click on Provision certificate
You will need to wait for the DNS update to enjoy your site in https

Deploy on content change

One last interesting feature is the ability to rebuild the website when content changes. As of now, if we add a new section on contentful, it's not reflected on the website. Let's change that so that any modification on the site triggers a rebuild of the website:

First head to the deployment settings in netlify, scrolling to build hooks
Clicking Add build hook, configure the name of your hook and the deployed branch:
Once saved, you need to copy the URL that you need to use to trigger the build:
Then go back to Contentful, going to webhooks in settings:
Click: Add a new hook
Name the hook as you like, then paste the netlify hook URL in the URL field:
Then you can either trigger the build for all event but I recommend limiting the hook to publish/unpublish actions to avoid useless builds.
Once the hook is saved, creating a new section will trigger a build on netlify:

Wrap-up words

Building a showcase website is mostly about content, presenting the products, services, the company goal/mindset. In this regard, using a CMS is the best way to enable your client to update their contents, keeping them happy

This article, was a simple introduction, to the currently available technologies enabling developers to build such a site. There are much more great features enabled by those tools, namely:

Working with images, videos, ...
Using different translations for contents
Workings with formatted texts without allowing contributors to inject html in the website

Thank you for getting this far and happy coding!

Don't lose sight of your bugs: How to improve your defect capture by 20%

Tue, 10 Mar 2020 00:00:00 GMT

Here we assume you correctly set up Sentry. Both your front-end and your back-end are able to send errors to Sentry and you set up Sentry releases (https://docs.sentry.io/workflow/releases/?platform=javascript).

Checkpoints

This is what we'll want to achieve

ALL unexpected behaviours (bugs) are captured in Sentry: no false negative
No expected behaviours are captured in Sentry: no false positive
Sentry issues contain all data needed to debug the issue

The first point is about identifying all bugs, the second about avoiding noise. This is all about making the workplace clean, uncluttered, safe, and well organized to help reduce waste and optimize productivity: see 5s.

Capture ALL exceptions

Why aren't all exceptions caught

Out of the box, Sentry works by listening to uncaught exceptions in your application. In addition, there are some integrations to listen to some technology-specific errors: eg redux-sentry-middleware.

Something to be aware of is that Sentry does not automatically capture caught exceptions: if you wrote a try/catch without re-throwing an exception, this exception will never appear in Sentry. Example:

async function getTodoList() {
  try {
    return await axios.get('/todos')
  } catch (error) {
    toaster.error('Error while retrieving todos')
    // `error` will never appear in Sentry! The only way for you to
    // know that this bug happens is waiting for users to report it,
    // and even then you'll have no data to debug it
    return []
  }
}

This is called "silencing" or "swallowing" exceptions.

Manually capture exceptions

The user experience of the code snippet above is fine: we show a toaster and do not crash the entire application. The only problem is with reporting: we need to know that this bug happened.

Sentry gives you an easy tool for this, that allows you to manually capture exceptions:

async function getTodoList() {
  try {
    return await axios.get('/todos')
  } catch (error) {
    Sentry.captureException(error) // this exception will now show up on Sentry
    toaster.error('Error while retrieving todos')
    return []
  }
}

Summary: exception handling rule

When you catch an exception, you have 2 exclusive choices:

re-throw it (throw error in Javascript)
Sentry.captureException

If this exception is a "normal" behaviour, make it clear with a comment. This should be very rare, you shouldn't use exceptions as "normal" control flow! But Python/Django sometimes do not leave you much choice, sadly.

Capture exceptions at the right place

This mostly applies to backend.

When an exception happens, it's usually quite deep down the call stack: for example accessing the database in a Django model's method called by a service called by a Django view. You want all exceptions to be captured, so you have 4 options here:

Let the exception propagate until it crashes your request (HTTP 500 internal error) or Celery task. That would be automatically caught by Sentry
Capture the exception as close to its source as possible: in the model's method
Capture the exception where there's most business logic context: in the service
Capture the exception as close to the app's edges as possible: in the Django view

Remember the exception handling rule: any exception caught should be either re-thrown, or manually captured. This means that you don't really have any choice:

An exception usually is due to an internal error: it should absolutely crash your request or your Celery task. It is important so that your logs and metrics are accurate, don't hide problems to yourself.
If for some reason you do not want the exception crashing your request/celery task (maybe it's something not-so-important that is acceptable to fail), you want to handle it as close to the edge of your application as possible: in the Django view for example. This allows all intermediate functions to have a chance to enrich the exception (raise SomeException from err in Python, or Sentry breadcrumbs) and is a good practice to avoid your core services making too much assumptions.

Remember that the exception handling rule is an either/or: you should not both capture the exception and re-raise it. The following is wrong:

def getById(id):
  try:
    User.objects.get(id=id)
  except Exception as e:
    Sentry.capture_exception(e) # This is wrong!
    raise UserNotFound from e # This is the exception that should be caught, higher in the call stack

It is wrong because it will lead to your exceptions being captured by Sentry multiple times, which means multiple Sentry issues for the same underlying problem, which means noise. Noise is bad, see 5S principles.

Capture non-exception bugs

Most bugs should trigger an exception (that you raise yourself if needed): invalid input, invalid state... But in some (hopefully rare) cases you might need to capture a bug in Sentry but still continue your work. For this, you can use Sentry.captureEvent() to capture fully-custom events.

This should be very rare: if you're in an invalid situation (bad input for example), probably you should raise an exception and let the caller handle it!

How to add extra data to Sentry events

Capturing the information is nice, but sometimes you have some data available in the code that you'd like to make available to Sentry for later debugging. Make extensive use of it: this makes the difference between an instant fix and a half-day debugging session because you can't reproduce a bug!

Breadcrumbs

Breadcrumbs are basically info logs. They are not errors themselves: they are only sent to Sentry when an error is actually captured. It allows you to get more info on what happened in the lifecycle of a request (backend) or in the user journey (frontend). You can probably assume that every log should also be a breadcrumb.

They are very easy to use, see the Sentry docs

function onLoginSubmit(event) {
  // ...
  Sentry.addBreadcrumb({
    category: 'auth',
    message: 'User logged in ' + user.email,
    level: Sentry.Severity.Info
  })
}

Context

You can set some "context" (set of key-value pairs, called "tags" or "extra data") to send to Sentry if and when an error is captured. For example, you might want to send the current user ID, the ID of the model targeted in a Django view, the referrer header, a tracing ID... For this you can use configureScope (https://docs.sentry.io/enriching-error-data/context/?platform=javascript#extra-context).

By default, the scope of the context is the entire request (backend) or lifetime of the app (frontend). You can set a scope local to a block of code (https://docs.sentry.io/enriching-error-data/scopes/?platform=javascript#local-scopes)

Extra data

Arbitrary key-value pair, that you can see on Sentry issues. You cannot filter issues based on these.

Example: value of local variable, message, timestamp...

Summary

Never swallow exceptions
Never captureException AND raise
In the backend, captureException as close to the edge as possible (Django views, Celery tasks)
Use configureScope extensively to add data to errors captured by Sentry

Master SQLAlchemy Relationships in a Performance Friendly Way

Mon, 09 Mar 2020 00:00:00 GMT

In a web application, performance is as important as any functional feature. In my current project, we experienced troubles with this aspect as we handled big databases with complex models and relationships. It worked fine when we tested the product with future users. But as soon as it got into production, the volume of data increased significantly. That's when the performance issues came out. We learned some useful stuff while fixing those issues. I will share them with you in this article.

First, each access to the database costs time, so you want to emit as few SQL statements as possible. The key to keeping this number stable when your data volume increases is to have a constant number of SQL statements emitted no matter the number of objects retrieved.

When handling objects with relationships, the query result is composed of nested objects and arrays. We learned the hard way that in those cases, serializing the query result to python object is an expensive operation. The easiest way to reduce the cost of serialization is to only retrieve the objects you need, nothing more.

Using the right loading technique for your relationships will help you reduce the number of SQL requests you emit. Also, it will allow you to retrieve only the relationships you need at the moment. Combined with better handling of query filters, it will allow you to reduce the serialization time of your queries.

1. Limit the number of queries emitted

SQLAlchemy provides several loading techniques to retrieve your relationships. You can specify the one you want to use when declaring a relationship, with the parameter lazy.

Loading techniques

relationship(ProjectModel, lazy="select")

There are 6 different loading methods. Here are, in my opinion, the 2 more useful :

Lazy loading: This technique is the default one. It will be used if you omit the lazy parameter in your relationship or if you specify lazy="select". This way of loading emits a separate query when first accessing the attribute.

class Parent(Base):
    __tablename__ = 'parent'

    id = Column(Integer, primary_key=True)
    child_id = Column(Integer, ForeignKey('child.id'))
    child = relationship("Child", lazy="select")

# emits a SELECT statement to get the parent object
parent = session.query(Parent).first()

# emits a second SELECT statement to get the child object
child = parent.child

Eager loading: The eager loading technique, also called joined loading, specify that you always want to retrieve the relationship. When accessing the parent element, SQLAlchemy will also join the relationship table. This way, when accessing the relationship attributes, it doesn't emit another SQL statement. The relationship is loaded when accessing the parent object.

class Parent(Base):
    __tablename__ = 'parent'

    id = Column(Integer, primary_key=True)
    child_id = Column(Integer, ForeignKey('child.id'))
    child = relationship("Child", lazy="joined")

# emits a SELECT statement to get the parent object and its children
parent = session.query(Parent).first()

# does not emit a second SELECT statement as child object is already loaded
child = parent.child

⚠️ If you want to use eager loading on a self-reference, you have to tell SQLAlchemy how many levels deep it should join. To do that, just add the parameter `join_depth` to the relationship declaration. Here is an example :

class Person(Base):
    __tablename__ = 'person'

    id = Column(Integer, primary_key=True)
    manager_id = Column(Integer, ForeignKey('person.id'))
    manager = relationship("Person", lazy="joined", join_depth=1)

person = session.query(Person).first()
person.manager # eager-loaded (one level deep)
person.manager.manager # lazy-loaded (two levels deep)

Choosing the right loading technique depends on when and how you use your relationships. As I said earlier, you want the number of requests to remain constant when the data volume evolves. That is why using the default loading technique can be bad for your app performance. If you retrieve multiple objects and their relationships, it is better to use the eager loading technique, as you will emit only one SELECT statement, no matter the number of objects you retrieve.

The trick here is that when developing a feature, you often manipulate a small number of objects. The lazy loading technique gets slow only when reaching a certain amount of data. So, measure the execution time with a realistic data volume. Also, if you know that the number of objects you manipulate will grow in time, you would better use the eager loading technique so the number of queries stays constant.

Be careful though not to use the eager loading technique too much. As I said before, retrieving useless objects is expensive. That's why I rarely choose this loading technique directly on the relationship declaration. I prefer to choose it on a query level, only when I'm sure I need the relationship. I will explain in the next section how you can achieve that.

2. Do not retrieve useless objects

As I said before, the serialization of query results in python objects is expensive. We noticed that executing the same SQL query on the database and with the ORM did not take the same time. It was faster on the database than with the ORM. The difference between the two is that the result on the database is raw data, while with the ORM you get nested objects. So we deduced that this was indeed the serialization that was increasing the request time.

There are two main things you can do to limit that :

Only retrieve object relationships when you need them. The deeper you go on the data you retrieve, the longer it will take to serialize the query result. You can do that by choosing your loading technique on a query level.
Do not wait to have python objects to filter the one you need. Likely, you can directly filter the data in the query, even if you need to filter by a relationship attribute.

To illustrate those two points, I will use the following example. You can see two models with one-to-one relationships, including a self-referential relationship.

class Person(Base):
    __tablename__ = 'person'
    id = Column(Integer, primary_key=True)
    firstname = Column(Text)
    lastname = Column(Text)

    address_id = Column(Integer, ForeignKey('address.id'))
    address = relationship('Address')

    manager_id = Column(Integer, ForeignKey(person.id))
    manager = relationship('Person')

class Address(Base):
    __tablename__ = 'address'
    id = Column(Integer, primary_key=True)
    number = Column(Integer)
    street = Column(Text)
    city = Column(Text)
    country = Column(Text)

Choose the loading technique on a query level

If you look at the example, you will notice that I did not specify a loading technique on the relationships. As I mentioned earlier, it is better to declare the loading technique in the query than in the relationship. This way, you can use different techniques depending on your needs. It will prevent you to retrieve relationships when you don't need them just because you set eager loading as the default technique.

Imagine you usually manipulate the Person object without their managers. The lazy loading is a good loading technique as you won't retrieve relationships you do not need. But then you want to return the list of all Persons with their managers for some purpose. You will want to eager load the managers with the persons so that you only make one SELECT statement. Here is how to do it :

Person.query.options(joinedload("manager"))

The joinedload option here has the same effect as if it was set in the relationship declaration. The difference is just that is will take effect only in one query, so you can use only if you need to get the relationship along with the main object.

The load options you can use in a query are the same that you can use in a relationship declaration. This allows you to have your default loading technique in the relationship but choose another in a query if needed.

Filter your queries by relationship attributes

Let's use the example from above. Imagine you want to get all the Person instances that live in Paris. You may get all the instances and then filter them with a python function. This is not the best way to do it. Filtering in the SQL query will be faster. First, because this language is optimized for filtering and other query operations. Also, the more objects you get, the longer it will take to serialize them to python objects.

Filtering by relationship attributes is quite simple. The only thing you have to do is to join your relationship before filtering, like this:

Person.query.join(Person.address).filter(Adress.city == 'Paris').all()

Now, you may encounter the need to filter on a self-referential relationship. To achieve that, you have to alias at least one side of your expression, so you can reference the table unambiguously. There are two ways of achieving that.

The first way is a bit verbose but clear and very flexible. You first define an alias for the table, here Person. By using it on the join, you will be able to explicitly tell which side of the relationship you are referring to when accessing attributes.

from sqlalchemy.orm import aliased

Manager = aliased(Person)
Person.query.filter(Person.firstname = 'Pauline').\
                join(Manager, Person.manager).\
                filter(Manager.firstname = 'Pierre').\
                all()

The second option is less verbose, but I find it confusing. In the join, you can use the property aliased. With that property, all use of the Person entity after the join will refer to the alias. You lose some flexibility as the sequencing of the functions is important here.

Person.query.filter(Person.firstname = 'Pauline').\
        join(Person.manager, aliased=True).\
        filter(Person.firstname = 'Pierre').\
        all()

Bonus

To help you improve your app performances, you may want to spot where you are lazy loading some relationships. If you use the raise loading strategy, with lazy="raise" in your relationship, SQLAlchemy will raise an error each time you try to retrieve a relationship without eager loading it first. This can be helpful to be sure you keep a constant number of SQL statements on big objects you manipulate.

Why my Lambda cannot access Internet anymore from its AWS VPC?

Mon, 20 Jan 2020 00:00:00 GMT

In this post, I try to be as clear as possible to explain how to give Internet Access to your AWS resources if they are in an AWS VPC.

For the most curious and those who like to understand things in depth, I explain at each step the AWS concepts in the grey part.

How did I get there?

Recently, I developed an application with a React frontend, a NestJS backend and I set up a deployment with an AWS Lambda function for the backend and an AWS S3 bucket for the static frontend. My app was quite simple and mostly used a PostgreSQL database, an external API and the Google authentication.

All was working really fine in production until I decided to switch my RDS database to private mode for security reasons and then lost its access. Indeed, by default, a RDS database is in an AWS VPC and you can't access it, except if you are in the same VPC or if your RDS is public.

Thus, to retrieve access to my database, I was forced to move my AWS lambda function in the same VPC. Then came a new problem. When you move an AWS instance in a VPC, you lose the Internet access and therefore cannot anymore reach your instance from Internet and your instance cannot reach the Internet anymore.

1 - Create a VPC

A VPC (Virtual Private Cloud) is a virtual network in the cloud in which you can launch AWS resources. You have complete control over your virtual networking environment, including selection of your own private IP address range, creation of subnets and configuration of route tables and network gateways.

You have all interest in using VPC because it helps in aspects of cloud computing like privacy, security and preventing loss of proprietary data. Moreover, some instances like AWS relational databases, need to be in a VPC and the only way to access them is to be in the same VPC.

If you don't already have a VPC (virtual private cloud), you need to create one.

To do that, go to the Services tab, select VPC, click on Your VPCs on the left menu and then on Create VPC:

Choose the name of your VPC, for example my-wonderful-vpc
In the CIDR (class inter-domain routing) block input, choose a range of IPs addresses for your VPC, for example 172.30.0.0/16

For your information, 172.30.0.0/16 is what we called a network mask, it means all the IP addresses starting from 173.30.0.0 to 172.30.255.255.

2 - Create private and public subnets in your VPC

A subnet is simply a range of IP addresses in the VPC. A subnet can be thought of as dividing a large network into smaller networks. This is done because the maintenance of smaller networks is easier and it also provides security to the network from other networks.

To host your lambda, you need to create a private subnet inside your VPC.

Click on Subnets on the left menu in the VPC service and then on the button Create subnet:

Choose the name of your subnet, for example my-wonderful-vpc-private-subnet
Choose the VPC you created during the previous step (my-wonderful-vpc)
In the CIDR block input, choose a subrange IPs addresses of your VPC IPs addresses, for example 172.30.1.0/24 (from 173.30.1.0 to 172.30.1.25)

Repeat these previous steps to create the public subnet by choosing another name (for example my-wonderfull-vpc-public subnet) and another subrange of IPs (for example 172.30.2.0/24).

You will configure your subnets as public and private in the 4th step.

3 - Create an Internet Gateway and a NAT Gateway in the VPC

An Internet Gateway is a logical connection between an AWS VPC and the Internet. It's not a physical device. Only one can be associated with each VPC. If a VPC doesn't have an Internet Gateway, then the resources cannot be accessed from the Internet. Conversely, resources within your VPC need an Internet Gateway to access the Internet.

A Network Address Translation (NAT) enable instances in a private subnet to connect to the Internet, but prevent the Internet from initiating a connection with those instances. To do that, NAT maps all the private IP addresses assigned to the instances in the subnet to one public IPv4 address called the Elastic IP address.

To access Internet, you will need to attach an Internet Gateway to your VPC.

Select Internet Gateways on the left menu and then click on the button Create internet gateway:

Choose the name of your Internet Gateway, for example my-wonderful-vpc-igw

You will then need to attach your Internet Gateway to your VPC. Come back to the Internet Gateways tab, select your Internet Gateway (my-wonderful-vpc-igw), click on Action, select Attach to VPC and select your VPC (my-wonderful-vpc).

You will also need to create a NAT Gateway. Click on NAT Gateways on the left menu and then on the button Create NAT gateway :

Choose the subnet you want to be public (my-vonderful-vpc-public-subnet)
Choose one of your Elastic IPs. If you don't have one, click on Create New EIP

Your infra should look like this now:

4 - Associate the right route tables to the subnets

A route table contains a set of rules called routes which determine where traffic has to be directed. You can create as many route tables in a VPC as you want. Route tables act at the subnet level, not the VPC level. A route table can be associated to one or severals subnets. By default, all route tables in a VPC have a local route for communication within the VPC. You can add custom routes in a route table by creating a new route defining which traffic (IP destination) must go where (target).

Remarks:

Even if you don't create any route table, all VPC come with a default main route table and all the subnets of the VPC are associated to this main route table until you associate them to a custom route table.
If a subnet is associated to a route table redirecting all traffic to an internet gateway, it is called a public subnet.

We will create two custom route tables, one for each subnet.

In the VPC service, click on Route Tables in the left menu and then on the button Create route table:

Choose the name for your route tables, for example my-wonderful-vpc-public-route-table and my-wonderful-vpc-private-route-table
Choose the VPC created in step 1 (my-wonderful-vpc)

Now that the two routes tables, we are going to configure them. Let's start with the public one.

Go back to the Route Tables section and select the created route table my-wonderful-vpc-public-route-table:

Click on the tab Routes of the selected route table
Click on the button Edit routes
Add a new route with for destination 0.0.0.0/0 and for target your internet gateway ID igw-.... created in step 3 and click on Save Routes

Then, click on the tab Subnet Associations of the selected route table
Click on the button Edit subnet associations
Add your public subnet my-wonderful-vpc-public-subnet created in step 2 and click on Save

By doing that, you have redirected all outgoing traffic of the public subnet to the internet gateway, what makes this subnet a public subnet.

Now, let's configure the private route table. Go back to the Route Tables tab and select the route table my-wonderful-vpc-private-route-table:

Click on the tab Routes of the selected route table
Click on the button Edit routes
Add a new route with for destination 0.0.0.0/0 and for target your NAT gateway ID nat-.... created in step 3 and click on Save Routes
Click on the tab Subnet Associations of the selected route table
Click on the button Edit subnet associations
Add your private subnet my-wonderful-vpc-private-subnet created in step 2 and click on Save

By doing that, you have redirected all traffic of the private subnet to the NAT gateway.

5 - Create the lambda function and configure it

The hardest part is already done!

If you don't already have a lambda, go to the Lambda service, and then click on the button Create function:

Choose a name for your Lambda, for example my-wonderful-lambda
Select the Node.js runtime
In the section Permissions, select the choice create a new role with basic lambda permissions

Once your lambda is created, click on it in the Lambda service and it will open the configuration page. Go down to the section Network and there, you must select 3 things:

your VPC created in step 1 (my-wonderful-vpc)
your private subnet created in step 2 (my-wonderful-vpc-private-subnet)
the default security group of your VPC, called by default your-vpc-name-default-security-group

🔥🔥 Congrats, that's all, your AWS lambda function has access to Internet! 🔥🔥

The end of this post is for people who wonders what is this mysterious default security group that I mentioned in the last step.

Bonus: Add more security with the security groups

A security group acts as a virtual firewall for your instance to control inbound and outbound traffic. You can create as many security groups in a VPC as you want. Security groups act at the instance level, not the subnet level. A security group can be associated to one or severals instances. By default, all security groups in a VPC allow all inbound and outbound traffic. You can add custom rules in a security group that control the inbound and outbound traffic.

Remark:

even if you don't create security groups, all VPC come with a default security group and all the instances of the VPC are associated to this default security group until you associate them to a custom security group

A good practice is to associate with each of your instance a security group with the strict minimum authorized traffic.

For example, if you have a RDS database (AWS relational database service), you will only need to accept inbound traffic from your lambda (or your other backend instance).

To do that, in the VPC service, click on Security Groups and then on the button Create security group.

Once created, go back to the Security Group tab, select your security group, go the Inbound Rules tab, and select Edit rules to add an inbound rule that allows only incoming traffic from the security group of your lambda instance to reach the RDS instance:

Then, go to the Outbound Rules tab, click on Edit rules to add an outbound rule that allows all traffic to leave the instance:

To finish, you just have to associate this security group to your RDS in its configuration page.

🔥 Congrats, that's all you have to do to protect your database from the outside.

I hope you have enjoyed and have learned new things in this post. Feel free to leave comments to help me improve it.

How to Synchronize your Style Guide from Figma to CSS in One Click

Fri, 17 Jan 2020 00:00:00 GMT

I have set-up an automatic pipeline to share our design system's style guide:

from Figma with a homemade plugin on which our designers work
to our web apps on which developers work

I'd like to share with you why I think this is awesome and how it will improve our collaboration. Then I'll get into the technical details and explain how I set this up.

A bit of context

With one of our clients, we are working on standardizing styles across our different apps and creating a design system. We have several web projects:

a landing page
a web app where users can see their account info, manage it
an extension where they have quick access to their info

Because these projects live on their own, design can get a bit hectic. Colors are similar but not quite the same, buttons have different looks...

That's why we created a shared design library consisting of:

shared components, with ready to use buttons, form elements that don't need to be developed on each project
a style guide defining the look and feel of the brand with colors, typography styles that should be used in all projects

But once we had these, updating the style guide was a lot of repetitive work. That's why I automated this process. Here is how it goes:

On Figma, the designer adds a color or updates it
Thanks to our homemade plugin, they send their changes to Github in one click in Figma
A script is triggered automatically, generating automatically new files that can be used on all web projects
A developer approves and merges the changes.
They publish the shared library new version.
Other projects developers upgrade their package. Colors are updated

Why share a style guide?

Creating a design system ensures that your product has consistency. This will strengthen your brand identity to your user and consistency will make it easier for the user to navigate through your familiar patterns. For instance, when submit buttons are always blue, the user knows what they do.

To implement this standard system in your code, you can centralize all your style variables in a stylesheet. Thus, you will improve your productivity by not having to decide what color to use. You don't have to use your color picker to figure out what color this text should be or end up with 50 shades of blue in your project.

Why make this automatic?

Making this automatic makes things faster, more reliable. When you don't spend time making tedious and repetitive chores, you have more time to focus your mind on solving problems and creating value.

Mostly it's a great way to give ownership to designers over the style guide and its implementation. Unlike a regular process where they ask for a change, wait for the team to take the time to do those changes, here they push the changes and decide when it will be done.

How did I do this?

I wanted to share color from Figma via the shared design system library to every web projects in CSS format.

From a technical point of view, here is what I've done to implement this process:

Develop a Figma plugin that:
- fetches the current colors on the shared component repository,
- compares them to the project's colors and displays the difference to the designer
- opens a Pull Request thanks to the Github API
Write a script with Handlebars templates and plop that creates a CSS file with color variables that can be used across all of our projects
Launch this script in the repository's CI so when the Pull Request is opened, the resources are created also.

Creating the Figma plugin

The Figma plugin aims to automatically export a project's colors to the shared design system repository.

It shows the differences being updated and once the designer has validated them, opens a pull request on Github.

Building this plugin was made easy thanks to Figma's documentation and automatic plugin generation tool directly integrated into the desktop app.

If you want to create your plugin, I have a few tips that could help you.

From your desktop app, in the plugin tab, you can create a one by choosing one of the templates:

an empty project
a project with no UI
a project with a UI

To choose between the last two, you may need to understand how a Figma plugin works. There are two parts: a sandbox and an iFrame

The background has access to Figma's controller, this is where you can get information on a project, its components and change it. This controller is very well documented on Figma's developer site and strongly typed if you are using Typescript.

The iFrame doesn't have access to the controller but it has access to browser APIs.

If you don't need any user inputs or feedback, you will only have a code sandbox but you won't be able to access useful APIs or make HTTP requests

Source: Figma Developers https://www.figma.com/plugin-docs/how-plugins-run/

In this case, I chose to create a Plugin with a UI.

In the sandbox, I get the projects "paints" (which can be colors, gradient or images) and only keep the "SOLID" paints (the colors) :

const paintStyles = figma.getLocalPaintStyles();
const solidPaintStyles = paintStyles.filter(
  paintStyle => paintStyle.paints[0].type === "SOLID"
);

I then create an object containing the color names, variants, and values.

In the UI, I push a new commit to the repository with this object in JSON format to Github API endpoint. I have a function that does this API call:

    const updateColorsJsonContent = async (
      masterSha,
      newColorObject,
      branch,
      token,
      userName,
      userEmail
    ) =>
      await fetch(`https://api.github.com/repos/our-repo/contents/src/globals/colors.json`, {
        method: "PUT",
        headers: {
          Authorization: `token ${config.token}`
        }
        body: {
          message: "feat(sync): applying Figma colors update",
          content: window.btoa(JSON.stringify(newColor, null, 2)),
          branch,
          committer: {
            name: userName,
            email: userEmail
          },
          sha: masterSha
        }
      });

Finally, I open a new pull request so a CI build will be triggered and a developer can review and approve.

Creating shareable stylesheets

I share design alongside components in our design system, repository. In this library, I have decided to store colors in a JSON file.

I chose this format on the shared design library because it was an easy format to export.

But our web projects mostly use CSS for style. To bypass this, I use the CI to generate a stylesheet with new color variables.

I use Handlebars Template to generate this CSS style guide. Adding a color to a CSS file would look something like:

:root {
  --{{color-name}}: {{value}}
}

If I pass the template a color name "blue" and a value "#aaa", I would get:

:root {
  --blue: #aaa;
}

In our case, our input is more complicated. Designers choose one main shade and derive it to several shades. It is an object shaped like this:

{
  "blue": {
    "1": "#aaa",
    "2": "#bbb"
  },
  "red": {
    "1": "#ccc",
    "2": "#ddd"
  }
}

To loop through each color and then through each variant, our template actually looks like this:

:root {
{{#each colorToken}} // loop through the color names
    {{#each this}} // loop through the color variants
    --{{@../key}}-{{@key}}: {{this}}; // add a color variable
    {{/each}}

{{/each}}
}

key and this refers to the current loop object key and value, in our case the color variant and value.

@../key refers to the parent loop key, the color name here.

As an output I get:

:root {
  --blue-1: #aaa;
  --blue-2: #bbb;

  --red-1: #ccc;
  --red-2: #ddd;
}

From there, it is easy to create a similar template and generate a variable file in SCSS.

To launch our templating process automatically, I use "plop" which is a command-line tool to call Handlebar from your terminal. I am now ready to update automatically this resource in our CI when a change is detected on the colors.json file, i.e. when the designer has created a pull request with updated colors from Figma.

I chose the CI to execute this because:

I am sure the CSS resources will always be up-to-date with the JSON
I don't want to overload our publishing process by adding this task
I don't want to do it in the plugin because I try to limit the logic as much as possible. The format which is used in the web project is very specific and can change. It is easier to deploy a new CI configuration than a new plugin.

Conclusion

I think we have a great process that will reduce the time for the designers to see their work deployed. I'll see if our team of developers and designers enjoy the time gain and simplicity.

Automation is a great way for developers to collaborate with designers. If you are interested in that topic here are a few tools you should check out:

Style Dictionary: an open-source project developed by Amazon that allows you to export your style variables into cross-platform formats for iOS, Android or web and maintain your style guide easily.
Overlay: a tool we develop at Theodo, that allows a designer to export their components from Sketch and will generate React and Vue code.

I hope this article will make you want to create your design pipeline and work more closely with your designer!

As A Developer, I know how to talk to my PO about existing code

Mon, 06 Jan 2020 00:00:00 GMT

Cover photo by You X Ventures on Unsplash

Monday, 10am. You're around a table with a new team, looking at a product owner presenting the project you'll be working on for the next few months.

While this project wouldn't be the core software of your company, it would play an integral part in the firm’s development over the next few years. The PO wants it to be done quickly and cleanly so that you can all get back to improving the main product. To help achieve that goal, they've found an old prototype that had been in a drawer for a few months - or even better, an open source solution doing something similar.

This should be fairly easy then: take the existing software, add what we need, and deliver.

tl;dr

There are many different ways we can find value in existing code, like product inspiration or architecture ideas.

Lean principles should not be forgotten when dealing with old or open source code.

Communication may become more difficult. We offer a few tips and a milestone-based feature/cost analysis tool to ensure everyone is on the same page.

A few thoughts to start with

Existing software has a lot of value outside of code itself. When approached carefully, it can provide valuable product insights and inspiration. It will prevent you from repeating mistakes, give you industry knowledge that you may not already possess... And even the code is useful, since you can let it inspire you architecturally, without the baggage of style of language specificities.
Development resources and available skills can drastically impact your plans. For example, we once had a project based on existing Java code but no Java developer on the team. The decisions we made were obviously influenced by that fact and a different team with Java developers might have reached a different conclusion. Both could be equally valid! You should help your Product Owner by considering alternative options, for example, "bringing in a contractor with X experience could reduce development time by Y."
Defining concrete, near-future milestones helps us focus on the short term and make tough decisions, which is especially important in a startup environment.
Existing code sets expectations and introduces bias in the way we think about the project. This clouds judgment and you may find the team struggling to communicate effectively and agree on the best way forward. This article will dig deeper into this issue and offer a few tools to help your team address it.

Human see, human want

Thanks to agile, everyone nowadays pays at least lip service to the notion that code only has value if it serves the user. It's the reason we work with stories and try to keep our sprints short: the more user feedback we can incorporate in our process, the less worthless code we produce.

I've noticed that all this tends to be disregarded when trying to incorporate existing code. Both the devs and the PO like shiny new things.

The PO sees a cool feature in a demo and all of a sudden it's a requirement in your project. You thought you were in for a simple time tracking app? Here come Face ID, SSO, custom transitions and a full-blown local database.

Devs easily adopt a black and white approach. "We'll just incorporate the open source code, it's free" and "the old code is just too bad, I have to redo it from scratch" are two sentences that I've said many many times, often based on my personal preferences for the technologies used. This can bias the opinion of the unsuspecting PO and deteriorate your relationship with them when your actual speed differs greatly from the planned one.

For example, if you’re looking at some oddly written code in a language you’re not familiar with, you might overestimate threefold the complexity of implementing it. Your PO, who’s used to you not overshooting my more than 50%, might interpret this as intentional and trust you less for it.

On the other end of the spectrum, if you promise something will be “free” (never a good move) and you end up spending a week on it, other features will be delayed and the whole project will suffer as a result.

So how can we align everyone’s expectations and settle on a common analysis of the situation?

Back to the drawing board

The next step is a very classic one: write user stories. Forget about the "free" existing code, just define your product as you would a new one. Depending on your company and team culture, this might be a task that only the PO works on but outside input - especially from user-facing stakeholders, or even better, users themselves - is always welcome.

The one question that should be on everyone's mind during the prioritization phase should be "Do OUR users actually want this or are we just copying the other product?". It may sound silly, but I've found double-checking this point saved us a lot of time. The other question to ask is “could we release WITHOUT this feature?" or "could the user benefit from the rest, even without this feature?". This forces prioritization. I've seen features originally touted as critical continuously fall deeper down in the backlog, and eventually be dropped altogether just by asking this question.

The prioritization should also be carried a little more into the future than usual, because we'll want to be able to assess the potential impact of using existing code in the medium term. If you usually work with weekly sprints and two weeks of backlog, you may miss out on benefits from the existing code by failing to foresee the need for some features or architectural choices.

For example, on one of our projects, extending our rough planning to two months allowed us to look into versioning and realize that the open source code we were adapting had a clever, working versioning system that we could just copy and paste if we designed our objects in a certain way. As a rule of thumb, staying close to the existing code will give you some options for the future, which have value.

Try to define some frequent milestones (maybe once a month, depending on the scope of your project) and define the specs of the product at that point. You may want to get a bit deeper than usual into their definitions and the technical strategies to achieve them. This will help us use the tools demonstrated later in this article.

In particular, you should have a decent idea of how your new code might interact with the existing one: are you going to rewrite an API? Use the existing one? Are you familiar with it? What would it take for you to be able to write this feature right now?

But wait, didn't we have existing code?

Yes we did! Now is the time to get back to it.

An existing chunk of code will have two effects on your work to reach a milestone:

It will help you complete some user stories at a lower cost. If the outside code does exactly what you want, you'll only have to spend a little time fitting it into your code. If it does not, then this will cost you some tweaks but the overall cost should be lower than writing everything from scratch.
It will also slow you down on everything else. Even if it's only a little, the architecture or style compromises that you will have to make will have an adverse impact on your overall speed.

The secret is to be able to quantify the effects. It is evidently impossible, but you will have to get a good enough estimate to decide which strategy to choose. On larger projects where the potential to integrate a lot of existing code exists, it might be worth getting real life data by actually integrating a feature from existing code and analyzing your execution speed.

Another useful tool to plan work is the milestones we previously defined. They can help us analyze the impact of existing code on time and feature constraints (a.k.a. deliverables). Below is a graphical visualization that we used in one of our projects.

To reach milestone 1, we have to develop 3 features: bootstrapping the project, implementing a basic auth and the Facebook one. Since we already have the first two in the existing code (and we consider the integration cost to be zero for simplicity), they do not require additional time or money.

Under the y-axis are the features from the existing code not yet integrated into our product.

To complete milestone 1, we have to build the Facebook integration. To do this, we spend time and money with the first derivative of the curve being our development speed or development cost.

We then start milestone 2, for which we can take the CMS integration from the existing code.

We repeat the process for all milestones and all features.

The constraints in this graph are the (x,y) coordinates of every milestone. They represent time / feature coupling that you have to respect for corporate and/or other reasons. They could be demos, client deployments, etc. In the end, they're the only thing you need to account for.

What's next?

We've only touched one of several aspects of working with existing code. I believe the most important thing in this situation is to keep a common language amongst the team that you can use to have productive discussions. In my experience, the most frustrating disagreements happen when one member of the team struggles to their ideas to the others. In these situations, we leverage the tools presented in this article to help us stay on track and be more productive.

Keep in mind that this is just an inspiration. Design your own tools! Your team’s needs will vary with the different personalities and skillsets involved, requiring bespoke solutions.

Github Actions on the test rig

Thu, 26 Dec 2019 00:00:00 GMT

As of November 13, 2019, GitHub Actions became publicly available. We have tested the automation solution from GitHub, often sold as a new CI/CD pipeline alternative. Let's see how it fares compared to its competitors, and whether you should switch to it.

1. What GitHub Actions brings to the table

A new approach to building your CI/CD pipeline

This new product is quite different from the features of common CI/CD providers (GitLab CI/CD, CircleCI, Travis CI...). Instead of offering only virtual machines for you to run your pipeline on, GitHub chose to offer both the virtual machines and pre-built blocks of CI/CD pipelines, called Actions. The goal of an Action is to save you from coding parts of your pipeline. They can be picked up in the marketplace or you can build your own, and reuse them on several of your repositories.

These can be common actions you would find in any Pipeline (Linting, Jest testing, Uploading items to an S3 bucket), and are almost plug and play. Other Actions could replace the sometimes frustrating WebHooks:

Updating your Codecov.io coverage
Automatically merging / asking for review on / tagging your PR
Sending Slack notifications
And more

Those actions are built by the community and are open source. You can fork them at will or create your own.

This system of sharing parts of a CI/CD pipeline also exists in other CI Providers (e.g. CircleCI Orbs), but GitHub took the "sharing" part to the next level by allowing you to share actions via their Marketplace, via Docker images, or via other GitHub repositories.

A competitive pricing

Another strong selling point for GitHub Actions is its pricing. They offer 20 free parallel builds and unlimited build minutes for every open source project.

For private repositories, with the free plan you will get 2'000 free minutes (~33 hours) of build time per month, on par with GitLab free tier. Compared to the 250 free minutes of CircleCI, or 100 free builds of TravisCI, GitHub Actions and GitLab are ahead of their competitors on the free tier offer. Just like GitLab CI, you can also set up GitHub Actions on premise for free.

If you have subscribed to a paid plan, the number of minutes is even higher (up to 50'000 minutes per month for the Enterprise plan).

And a great integration with other GitHub services

One thing we loved while playing with GitHub Actions is the smooth integration it offers with other GitHub features. You can trigger Actions and Workflows with pretty much any GitHub event: a classic opening of a PR, a push of a new commit, but also an issue opening, a comment, a PR review... you get the point.

You also get an easy access to the full context of the PR (or issue, etc.) which triggered the action, and to the JavaScript GitHub API thanks to the github-script action, which lets you write data back to GitHub. These two combined allow for a wide range of automation possibilities.

2. GitHub Actions drawbacks and teething issues

GitHub Actions still has a few limitations compared to its competitors, which makes it less adapted to heavy CI/CD pipelines. But the GitHub team is closely listening to its users and is rapidly bridging the gap: for example, the much-wanted dependency caching feature was developed just before the end of GitHub Actions beta phase.

No availability for legacy pricing plans

If your organization has decided to stick to legacy pricing plans because you get more bang for the buck from them, then you're out of luck: GitHub Actions will simply not run on any of your repos, even the public ones.

Performance of similar operations falls behind other providers

We tested GitHub Actions against CircleCI, and we found the same operations were 30% to 50% slower on GitHub Actions compared to CircleCI (though we couldn't fathom why). Here are tests we ran:

Setup

CircleCI: default Docker executor of free plan (medium size: 2 vCPUs, 4GB RAM)
Github Actions: Default machine (at the time of testing: 2-core CPU, 7GB RAM)
Image for both: Ubuntu 18.04
No caching

Results

Project	Command	Avg Github Actions duration	Avg CircleCI duration
yarn	`yarn && yarn build && yarn lint`	1 min 32 s	0 min 52 s
prettier	`yarn install`	0 min 37 s	0 min 18 s
prettier	`yarn lint`	0 min 31 s	0 min 15 s
prettier	`yarn test`	4 min 02 s	3 min 07 s

No schematic view of workflows

Mature CI providers such as CircleCI usually provide a schematic view of workflows that helps you know if you've assembled your jobs well. GitHub Actions doesn't provide such a view for the moment.

Additionally, the UI is lacking a few features such as branch or event filtering.

A view of a CircleCI workflow. Hopefully you won't get these jobs tangled like your earphone cords.

Note: workflow view was previously available when GitHub Actions were in beta, but was removed when GitHub switched to YAML-based workflow.

A few glitches here and there

During our tests, we noticed a few workflows that would be stuck at startup or would not be displayed in the interface, or their loader would keep spinning even when the jobs were finished. Possibly, these are simply teething problems that the GitHub team will soon take care of.

File caching is limited to 400 MB per individual cache (after compression) - UPDATED

Updated: on January 6th, 2020, the GitHub Actions team raised the per-cache limit to 2 GB, solving most people's needs. Original article text below:

In a CI pipeline, caches are usually used to quickly fetch builds or dependencies that you've installed in a previous CI run, to speed up future CI runs. However, many users are finding the 400mb limit too small for their uses, notably for those big node_modules folders.
At the time of writing though, the GitHub team is considering increasing this limit.

3. So, what should GitHub Actions be used for?

👍 Yay for light CIs, or for parts of your development workflows that are strongly integrated with GitHub.

🙅 Nay for heavy workflows, you may want to stick to (or start with) a more mature CI provider for now. GitHub Actions is however rapidly improving, you'd better keep an eye on it.

How to Integrate Typescript with Vue.Js

Thu, 21 Nov 2019 00:00:00 GMT

A few weeks ago I started a new Vue project. One of my co-dev recommended using TypeScript on our project: "It will help us spot bugs and mistakes, and the sooner we add it, the easier it will be". It was my first time developing on TypeScript.

Why TypeScript ?

TypeScript homepage

First of all, you should now what TypeScript is doing. TypeScript is a language that has the same syntax as Javascript. TypeScript is what we call a superset of Javascript. This means that every Javascript code you write will be a valid TypeScript code, and can be compiled by TypeScript. Therefore, you can add TypeScript to your already existing Javascript project. TypeScript’s main goal is to provide optional static typing and type inference.

// The variable x will be seen as a boolean
let x: boolean

// Here the type inference makes y as a boolean too
let y = true

You can type variables, function parameters, the function returns... TypeScript will then do a static analysis of your code to check dangerous operations (in terms of type). You can accidentally try to assign a variable with the wrong type or access a property of an undefined object. The last issue occurs a lot in run time, you didn't check that your object wasn't null. And there you go, your code crashes...

let x: boolean

/**
If you want to assign another a value with the wrong type 
TypeScript will trigger an error. In my editor I have : 
Assigned expression type "This is a string, not a bool" is not assignable to type boolean
**/
x = 'This is a string, not a bool'


/**
Here 'find' can return undefined if no correct document is found. 
Therefore, accessing the property 'type' will trigger a runtime error
**/
const documents = [{type: 'Review'}, {type: 'Book'}]
const myArticle = documents.find(document => (document.type === 'Article'))

const myArticleType = myArticle.type

With the last example I will have the following error during compilation :

Hope this introduction convinced you to use TypeScript. If you are new to it, I suggest reading the handbook

Let's now see how to install it on your Vue project.

Using TypeScript with Vue

Installation of TypeScript

On a new project

If you start a new project you can use the Vue CLI with your own settings and select Typescript in the choices.

Then say yes to use class-style component syntax. We will see later why you should use this syntax.

On an existing project

If you add it to an existing project, you can still add TypeScript with NPM :

npm install -g typescript

And you can check the recommanded configuration for your TypeScript config.

Code syntax to use TypeScript in Vue

First, let's tell our Vue compiler that Javascript will be TypeScript. In your Vue file, in the script tag, do the following :

<template>
</template>

<!--Add lang="ts" to specify it's TypeScript-->
<script lang="ts">
</script>

<style>
</style>

Then we need to modify our syntax to be TypeScript friendly.

During the installation (with Vue CLI) I suggested you use class-style component syntax. But other syntaxes exist. In Vue, there is 3 main syntax : the Options API, the Composition API, and the Class API. I suggest you use the latter. But we will see how to use TypeScript with them.

Options API

This syntax is the basic syntax of Vue. But you need to export your component differently. The usual export doesn't have type inference enabled :

<template>
</template>

<script lang="ts">
export default {
  //No type inference
  components: {},
  data() { return {} },
  computed: {},
  created() {},
  methods: {}
}
</script>

<style>
</style>

Therefore, you will need to use Vue.extend syntax to export your Javascript :

<template>
</template>

<script>
import Vue from 'vue';

// Note the Vue.extend
export default Vue.extend({
  //Type inference
  components: {},
  data() { return {} },
  computed: {},
  created() {},
  methods: {}
})
</script>

<style>
</style>

At the beginning of my project, we were using the options API for our Vue files. But we had some issues with TypeScript and decided to use Class API. Honestly, in hindsight, I think some issues came from us not using TypeScript correctly with the options API. Like for example, not typing functions returns. And now with the latest update of TypeScript I cannot reproduce these errors anymore.

But I still suggest you use the class-style component to code as you might, like me, have small issues with the Options API. Moreover, there are more example on the web with class-style components.

Composition API

This syntax is the new syntax of Vue 3. The syntax has been build with TypeScript integration in mind. Therefore, there will be better TypeScript support with the composition API and I think you won't have to change your syntax to use TypeScript. If you want to use this syntax in Vue 2, the composition-api package is available. Vue 3 will be released start of 2020 and is currently in (pre)-alpha. If you have time to take a look at this new syntax I suggest you to start coding with it to familiarise yourself with Vue 3.

But if you prefer to wait for the official release before using the composition API, then take a look at the last syntax available.

Class API

The primary goal of introducing the Class API was to provide an alternative API that comes with better TypeScript inference support.

To use the Class API syntax, you will need to install the package vue-class-component (already installed with Vue CLI).

Then create a Vue component using the following syntax :

import Vue from 'vue'
import Component from 'vue-class-component'

@Component({
  props: {
    myProp: {
      type: String
    }
  }
})
export default class HelloWorld extends Vue {
  myData: {name: string} = {name: 'test'}

  // computed are declared with get before a function
  get myComputed(): string {
    return this.myData.name
  }

  created() {}

  // methods are just functions
  myMethod(): boolean {
    return false
  }
}

You will get used to this syntax in no time!

One advantage of this syntax is that you can regroup 'methods' and 'computed' by functionality (the composition API is also built to do that).

import Vue from 'vue'
import Component from 'vue-class-component'

@Component({})
export default class HelloWorld extends Vue {
  // Autocomplete functionality
  get computedRelatedToAutocomplete() {
    ...
  }
  methodRelatedToAutocomplete() {
    ...
  }

  // Search Functionality
  get computedRelatedToSearch() {
  ...
  }
  methodRelatedToSearch() {
  ...
  }
}

The package vue-class-component is fully supported by Vue 2.

One more improvement we can make is by using the package vue-property-decorator. This package is also installed by Vue CLI.

With it, even the props will be inside the definition of the component :

import { Component, Prop, Vue } from 'vue-property-decorator';

@Component
export default class HelloWorld extends Vue {
  // The props are defined here with the annotation @Prop()
  @Prop()
  private msg!: string;

  myData: {name: string} = {name: 'test'}

  // computed are declared with get before a function
  get myComputed(): string {
    return this.myData.name
  }

  // methods are just functions
  myMethod(): boolean {
    return false
  }
}

Drawback : The Class API is still not perfect with TypeScript, and it will be replaced by the Composition API in Vue 3. After Vue 3 is released I strongly encourage you to use the Composition API.

But currently, it's our best solution! And changing from one notation to another is not that complicated. You can change one file at a time and still run the other files with another syntax. That is what we did to change from the options API to the class API. And even today, we still have old files with the options API syntax (that we are migrating little by little).

Actually, you can also implement TypeScript one file at the time with Vue (if you add it to an existing project). You take one of your Vue files and add the lang="ts" inside the script tag. Then you modify your component to use the Class API and fix the potential errors TypeScript found. It's really easy!

Tips and tricks

Type your props in Vue

In your project, you will want to type your props. Indeed, the type you define for your props should be one from Vue type. You can use Boolean, String, Array but no self-made type.

import Vue from 'vue';

type MySuperType = {
  name: string
  age: number
}

export default Vue.extend({
  props: {
    mySuperProps: {
      type: MySuperType, // Will not work
      required: true
    }
  }
})

You will need to use another syntax :

import Vue from 'vue';

type MySuperType = {
  name: string
  age: number
}

export default Vue.extend({
  props: {
    mySuperProps: {
      type: Object as () => MySuperType,
      required: true
    }
  }
})

Using Typescript with Vuex

If you plan to use Vuex in your project, then you can also Type your stores. In this article talking about Vuex and TypeScript you will find a detailed explanation on how to type your store and how to use the actions/state/getters inside a component with the Class API.

Access property on a nullable object

Sometimes you will have a nullable object. Before accessing the property of the object you will need to check if the object is not null. But you cannot do this check anywhere. It needs to be close to the access of the property.

Here are some examples/tricks for nullable objects :

import { Component, Prop, Vue } from 'vue-property-decorator';

@Component
export default class HelloWorld extends Vue {
  value: null | {test: string} = null
  setValue() {
    this.value = {test: 'hello'}
  }

  /**
   * this.value can be null,
   * we cannot access the property test on null
   *
   * An error will be triggered
   */
  methodWithTypeError() {
    this.value.test //error
  }

  /**
   * checking that value is not null before accessing property
   * will tell TypeScript that the variable is safe
   *
   * No error will be triggered
   */
  methodWithNoError() {
    if(this.value === null) {
      return
    }
    this.value.test
  }

  /**
   * When performing its analysis TypeScript will not check the whole code
   * Therefore, even if we have checked this.value before
   * it will still see it as null inside the map function
   *
   * An error will be triggered
   */
  methodWithErrorWhereItShouldBeOK() {
    {
      if(this.value === null) {
        return
      }
      const array = [1, 2, 3]
      array.map((arrayValue: number) => {
        return arrayValue + this.value.test //error
      })
    }
  }

  /**
   * Here by assigning this.value.test to
   * another variable just after the null test,
   * TypeScript will infer its type as not null
   *
   * No error will be triggered
   */
  fix() {
    {
      if(this.value === null) {
        return
      }
      let test = this.value.test
      const array = [1, 2, 3]
      array.map((arrayValue: number) => {
        return arrayValue + test
      })
    }
  }
}

Another solution for this kind of problem is to use the function get of lodash and have a default value returned if the object is null. But with this solution, you don't use TypeScript when accessing a property...

It took me some time to get used to TypeScript. In the beginning, I didn't understand the full strength of TypeScript and thought it was extra work. But as time went by, I started to get used to TypeScript and my Javascript code was more structured. It is now easier to develop, especially when coding along with other developers : TypeScript allows you to give a shape to your object, thus making them more understandable to others.

Hope this post convinced you to use TypeScript and that it was useful to you.

Thanks for your time!

Your React App Deserves a Proper Seo

Mon, 18 Nov 2019 00:00:00 GMT

SEO has become an integral part of a developer’s daily missions. Creating a robust, performant and well-designed web app is great, but useless if you don’t manage to get traffic on it. That’s where SEO gets in the game. In this article, you will understand why it is difficult to have a good SEO when creating a full app using React, and find solutions to overcome these obstacles.

What is SEO and why you should care about it?

Search Engine Optimization is a set of techniques which goal is to optimize the visibility of a website on the web search engines. Basically, when a search engine is looking for the best results according to a user’s search, it looks into a huge database called the index. The latter is filled by bots, the crawlers, which job is to walk around from site to site, analyze them and gather information in the index. The more optimized is your website, the better are the chances that it will appear in the top ranks of the results, which is crucial. Indeed, more than 60% of the traffic goes to the first three results of a web search. Thus, if you want your website to be visited, you should do your best to have a good ranking on search engines.

Why is it more difficult with React?

When creating a web app using React, you will face several issues concerning SEO.

Only one URL

React is implementing the concept of Single Page Application (SPA), so your application will have only one URL. This is absolutely not a problem if you want to create a showcase site. However, it will become one if you want to build a more complex app, with several virtual pages where users can navigate. You can simulate a page change by switching components conditionally to user’s interaction with your app, but it will remain the same page with the same URL. So, crawlers will only be able to read part of the application: the rendered components representing the first virtual page of the application when clicking on the unique URL. If each page had its own URL, crawlers could then be able to read your whole application page by page, and all content would have been seen.

All in Javascript

Speaking about crawlers… Search engines have many issues to render JavaScript (for the few that can actually do so), and React is before all a JavaScript library. A classic React application will be composed of an index.html file with barely any content in its body: a div which will be the entry point for React. Then, the components you created will be injected into this div, as JavaScript. So, if crawlers don’t manage to read the JavaScript, they could think that your whole application consists in a single empty div.

First solution: using React Router

One URL problem

The first problem I pointed out is that with React all your components will be rendered on a single page. With React Router, you will be able to do client-side rendering even deeper by creating a router on the client side.

Let’s see how it works with an example:

Example of navigation with React Router

When a user wants to go on the website, a request GET /index.html is done to the server.
The server sends the index.html page which contains scripts to launch React and React Router.
The application is loaded on the client-side.
The user clicks on a button to go on a new page /foo.
React Router intercepts the request before it goes to the server and handles the change of page itself, by updating the rendered components and changing the URL locally.

Thus, you will have several URLs.

However, there is another issue that appears now, as you are doing client-side routing. Can you see it? To understand well the next problem that we will face, let’s continue the previous example. The user saw the /foo page and wants to share it to his friend. So, he texts him saying: “Look at what I just found: go to my.awesome.page.com/foo”. The second person clicks on the link and arrives on a 404 page.

The second user arrives on a 404 error

Indeed, your server only knows the index.html page. Only React Router can understand the request GET my.awesome.app.com/foo once it is loaded on the client side.

So, with React Router you will allow your users to navigate in your application locally through internal links, but there is only one entry point to the application, which is your index.html. Every time you will request a page (which is not the index one) to the server, by following a link, refreshing or typing it directly in the URL bar, you will face a 404 error. But no panic, solutions exist!

Creating a catch all route

A simple way to handle this problem is to create a catch-all route on your server. The principle is quite simple: make it forwarding all requests to index.html. Once index.html is loaded, React Router will take over and handle the routing. More precisely, it should forward only the requests that the server doesn’t find because actually we don’t want the requests for media, for instance, to be redirected to index.html.

The server sends the index.html and React Router displays the foo page

For example, if you are on an Apache Server, you could write in your .htaccess this:

RewriteBase /
RewriteRule ^index\.html$ - [L]
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.html [L]

React Helmet

Now, you will have an app with several URLs, all working. That’s great. However, there is still a small step to follow for your pages to be fully independent. Indeed, you still have a unique index.html, so the same metadata for all your pages. Metadata is crucial for SEO, and you should have different metadata representing each page. What you can do is to use React Helmet. This package will help you to manage the metadata of each page of your application by adding React components into your code. This is really simple to implement.

Here is an example of how to use Helmets:

import React from "react";
import { Helmet } from "react-helmet";

class Application extends React.Component {
  render () {
    return (
        <div className="application">
            <Helmet>
                <meta charSet="utf-8" />
                <title>My awesome app</title>
                <link
                  rel="canonical"
                  href="my.awesome.app.com"
                />
            </Helmet>
            // the rest of your code here
        </div>
    );
  }
};

With a catch-all route along with React Helmet, your app is now composed of several pages, all working and independent!

Problem of Javascript

The first problem is now behind us. However, these different steps don’t change the fact that JavaScript is not well crawled. This problem can be tackled by using prerender.io. This tool will render your code in a browser and save the static HTML. When a crawler will try to visit your website, prerender will take over and send the HTML files, simplifying the understanding of the crawler.

In short, if you are using React Router, creating a catch-all route, using React Helmet and prerender will help you reaching your ranking’s goals!

Second solution: going isomorphic

Going isomorphic is a more complicated solution to set up if your project already exists and uses React Router, but it will improve your site in several ways: performances will be better, along with SEO.

Isomorphism is a term used to define an application which will use both client-side rendering and server-side rendering to take the best parts of each method. With server-side rendering, the application will be computed directly on the server, and a first rendered page will be sent to the client, in order to get the first content fast. This will improve performances as a server is generally more powerful than a mobile phone for instance. Scripts to launch React are then sent. The page is now interactive and the user can navigate, without requesting the server again, also improving performances.

If you want to go isomorphic, you don’t have to imagine a solution, it has already been created for you. You can use for instance Next.js, which is a React library that will provide you everything you need to create an isomorphic application. One requirement is to have a server that can execute Node.js.

The server computes a page, sends it to the user, which then can navigate on client side once React is loaded

This solution has a more important learning curve and is more complicated to set up than the first one, but if you want your app to be really fast and to have a perfect SEO, you really should give a try to Next.js.

Great! Whichever solution you have chosen, you tackled the one URL problem and crawlers will not be in trouble anymore reading your content. Your app is now ready to be in the first rank results. But there is still much to do in this regard and to help you ascend these ranks, I advise you to read this insightful article about how to rank first on Google.

I hope you enjoyed this read,

Need to encrypt your existing AWS RDS database? Follow the guide!

Wed, 13 Nov 2019 14:00:00 GMT

Let's begin by a little game: let me guess how you reached this article, OK?

You were working on a small project and wanted to deploy it on AWS, maybe it was your first cloud try. You configured basic database instance on RDS, and as there were a lot of ununderstandable options, you kept default configuration (really, who cares about configuring tools when only working on a POC?).

Then your project got bigger and you began to store more sensitive data. As you are quite aware of security issues, you wanted to check on the AWS console that your database was well encrypted.

But, hell! Your database was completely unencrypted! Moreover, you realized there was no option to encrypt your existing database on the fly...

Did I guess well :)?

No? Argh...

Well, if you're still reading, I must be close to the truth!

The problem here is that database encryption is not ticked on default RDS configuration. Moreover, depending on which instance size you chose during database instantiation, encryption was maybe simply not available at all!

Don't panic! In this walk-through guide, I will list all prerequisites needed for encryption on an RDS instance and cover all steps to follow to encrypt your database without losing any data.

Warning: as we will have to create a fresh new RDS instance to encrypt it, there will be an unavoidable downtime, whose duration will depend on your database size.

NB: We could tackle this trade-off by using a more complex technique (by using replica databases, and switching between main and backup databases during backup/restore operations covered in this tutorial). If you would like to see such a more advanced tutorial in the future, feel free to ask for it in comments ;).

Ready! Set! Go!

Prerequisites

First of all, your RDS instance must support encryption (Captain Obvious inside!). Almost all RDS instance classes support it, but you should double-check that your instance class is not one of the not compatible ones listed by AWS. If you are in that case, you must previously migrate your instance to another instance class that supports encryption (this migration can be done "in-place", but you should expect a downtime during migration).
You must also previously create an encryption key on the AWS KMS tool. It will be used by AWS to encrypt your RDS instance, so you should create a specific key for this use case. Be sure to create this key in the same zone as your database one, otherwise, it won't be available when configuring encryption.

OK, now you are ready to deep dive into database encryption!

We will start by backing up your existing database, then we will create a new RDS database and restore our backup file to it. Let's start!

Prepare your existing database for encryption

Here, we are going to back up our existing database and encrypt this snapshot during backup, using our previously generated KMS key. To reach this goal, follow these steps:

Log on the AWS console.
Reach RDS instances management interface (ensure to be in the right AWS zone) then select the database you want to encrypt.
Once on your instance configuration interface, on the top right, click on Actions menu, then select Take snapshot:

Give a name for this snapshot, then click on the Take Snapshot button:

Wait for the completion of snapshot creation (duration will depend on your database size).
Once completed, select the snapshot you just created to display its details:

On the top right, click on Actions menu, then select Copy Snapshot:

Give another snapshot name for this copy, select Enable encryption, then select the KMS key you previously created in Master key dropdown list, and click on Copy Snapshot:

This step is crucial for encryption: it will indicate AWS to create an encrypted database when restoring from this snapshot.

Head to snapshots list.
Wait for the completion of snapshot duplication (duration will depend on your database size: expect same duration as snapshot creation one).

Back up step is now cleared. We can then work on the restoration step and...

Create your new encrypted database

Here, we are going to create a fresh new encrypted database with data from our previous database by simply restoring our snapshot. As usual, simply follow these steps:

Go back to the RDS instances management interface then select your current database.
Once on your instance configuration interface, on the top right, click on Modify icon.
In the Settings section, edit instance name by modifying DB instance identifier input then click on Continue:

On the confirmation page, select Apply immediately then click on Modify DB Instance:

Wait for this modification to be complete.
From now on, your database is no more reachable with its previous ARN and its the start of your downtime.
Head to snapshots list.
Select the encrypted snapshot you just created to display its details:

On the top right, click on Actions menu, then select Restore Snapshot:

On the restoration page, fill all fields with the same values as your previous database (especially, DB Instance Identifier field must contain the name of your previous database before renaming it), then click on Restore DB Instance:

If you have been curious, you should have seen on the restoration page that the Encryption section was enabling encryption and was using your KMS key. All this information is directly related to settings contained in the snapshot you used for restoration:

Wait for the completion of snapshot restoration (duration will depend on your database size: expect 3x to 5x snapshot creation duration).
If your database was part of a specific VPC/Security group, double-check that your new instance is also configured the same way and modify it if needed.
Once finished you can check on the Configuration tab on your instance details page that your database is now encrypted:

If you want, you can now delete your former RDS instance to save some money.

And voila!

Your new RDS database is now totally encrypted, contains the same data as your former one and is reachable through the same ARN.

Well done!