<![CDATA[Theodo]]>https://blog.theodo.comGatsbyJSFri, 29 Apr 2022 13:00:16 GMT<![CDATA[Writing Quality Code Using the Chain of Responsibility Pattern]]>/2022/04/chain-of-responsibility-design-pattern//2022/04/chain-of-responsibility-design-pattern/Fri, 29 Apr 2022 00:00:00 GMT<style> a { font-weight: normal !important; } p > code { color: #eb5757 !important; background: rgba(135,131,120,0.15) !important; } code { line-height: normal !important; border-radius: 3px !important; padding: 0.2em 0.4em !important; font-size: 0.7rem !important; } </style> <p>At the beginning of my software engineering internship, I had little experience with writing clean code. The features were functional but there was hardly any underlying organization. I was lacking guidelines on what classes to create and how they should interact with each other. That’s when my mentor introduced me to design patterns. These are solutions for creating flexible and maintainable modules.</p> <p>In the upcoming sections, I will showcase the use of the Chain of Responsibility pattern and how it helped us write scalable and easily testable code on a production-level API.</p> <h2>A design problem</h2> <p>On a project for a renewable-energy provider, I was working on a consumption monitoring app. One goal was to retrieve the user’s activities over a time period. An <code>activity</code> is a carbon footprint calculated based on energy consumption. To compute the activities, we had to fetch and process the consumption in the following priority order:</p> <ul> <li>The <strong>real consumption</strong> (meter reading), retrieved from an external API</li> <li>An <strong>estimate of the household's annual consumption</strong>, retrieved from an external API</li> <li>An <strong>estimate calculated from the user's behavioral data</strong> (thereafter labeled macro estimate).</li> </ul> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/851c49d581128bcc11f715a92e1eb0ae/50383/data-fetch-flow.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 37.2972972972973%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAHCAYAAAAIy204AAAACXBIWXMAAAsTAAALEwEAmpwYAAAA/UlEQVQoz22RiYqEQAxE5///TBAVFARvwdvxvs8MlcFdZ2cDgY5Jql+1j7IsqW1bmqaJhmHgc9d1VBQFua5LjuNQ0zScdV1TVVXk+z4ZhkFZlvH81cf+A0LP55MF4jgmRVHItm2ukyThJdM0SZIkEgSBPM/7mU3TlCzLIlEUefY4jrfgPM+0LAv1fc90SHzD4jiOnCABIWZAA2fbtjExxDCzrutb8L+AoCzLbA0330PXdSaGeBRFTIgLz/P8FURxT4jAVp7nXz2QhmFI+74zIZ4IAKg/BK+4zvgpQRB8XIjAU6iqymfY1zSNe1+C97wWQfBXEFZBj8A+XMAR+i//oBeiKKgqiAAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Data fetching and processing flow diagram" title="Data fetching and processing flow diagram" src="/static/851c49d581128bcc11f715a92e1eb0ae/50383/data-fetch-flow.png" srcset="/static/851c49d581128bcc11f715a92e1eb0ae/1d79a/data-fetch-flow.png 185w, /static/851c49d581128bcc11f715a92e1eb0ae/1efb2/data-fetch-flow.png 370w, /static/851c49d581128bcc11f715a92e1eb0ae/50383/data-fetch-flow.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>The returned activities are the result of the first of these methods that returns a non-empty array.</p> <h3>A trivial solution</h3> <p>While designing a solution for the presented situation, we have to take into account the maintainability of the code, the simplicity of adding and removing data sources as well as testing our service. An initial solution may consist of cramming everything in one class. </p> <div class="gatsby-highlight" data-language="php"><pre class="language-php"><code class="language-php"><span class="token keyword">namespace</span> <span class="token package">Component<span class="token punctuation">\</span>Activity<span class="token punctuation">\</span>Provider</span><span class="token punctuation">;</span> <span class="token keyword">class</span> <span class="token class-name">ActivityProvider</span> <span class="token punctuation">{</span> <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">retrieveActivities</span><span class="token punctuation">(</span><span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span><span class="token punctuation">:</span> <span class="token keyword return-type">array</span> <span class="token punctuation">{</span> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">count</span><span class="token punctuation">(</span><span class="token function">retrieveMeterActivities</span><span class="token punctuation">(</span><span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">></span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token keyword">return</span> <span class="token function">retrieveMeterActivities</span><span class="token punctuation">(</span><span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">count</span><span class="token punctuation">(</span><span class="token function">retrieveEstimatedActivities</span><span class="token punctuation">(</span><span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">></span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token keyword">return</span> <span class="token function">retrieveEstimatedActivities</span><span class="token punctuation">(</span><span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">count</span><span class="token punctuation">(</span><span class="token function">retrieveMacroActivities</span><span class="token punctuation">(</span><span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">></span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token keyword">return</span> <span class="token function">retrieveMacroActivities</span><span class="token punctuation">(</span><span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token keyword">private</span> <span class="token keyword">function</span> <span class="token function">retrieveMeterActivities</span><span class="token punctuation">(</span><span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token variable">$data</span> <span class="token operator">=</span> <span class="token variable">$this</span><span class="token operator">-></span><span class="token function">fetchMeterData</span><span class="token punctuation">(</span><span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token variable">$this</span><span class="token operator">-></span><span class="token function">processMeterData</span><span class="token punctuation">(</span><span class="token variable">$data</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token keyword">private</span> <span class="token keyword">function</span> <span class="token function">fetchMeterData</span><span class="token punctuation">(</span><span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token comment">/* Fetch data */</span> <span class="token punctuation">}</span> <span class="token keyword">private</span> <span class="token keyword">function</span> <span class="token function">processMeterData</span><span class="token punctuation">(</span><span class="token keyword type-hint">array</span> <span class="token variable">$data</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token comment">/* Process raw meter data */</span> <span class="token punctuation">}</span> <span class="token keyword">private</span> <span class="token keyword">function</span> <span class="token function">retrieveEstimatedActivities</span><span class="token punctuation">(</span><span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token variable">$data</span> <span class="token operator">=</span> <span class="token variable">$this</span><span class="token operator">-></span><span class="token function">fetchHouseholdEstimatedData</span><span class="token punctuation">(</span><span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token variable">$this</span><span class="token operator">-></span><span class="token function">processHouseholdEstimatedData</span><span class="token punctuation">(</span><span class="token variable">$data</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token keyword">private</span> <span class="token keyword">function</span> <span class="token function">fetchHouseholdEstimatedData</span><span class="token punctuation">(</span><span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token comment">/* Fetch data */</span> <span class="token punctuation">}</span> <span class="token keyword">private</span> <span class="token keyword">function</span> <span class="token function">processHouseholdEstimatedData</span><span class="token punctuation">(</span><span class="token keyword type-hint">array</span> <span class="token variable">$data</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token comment">/* Process raw estimated data */</span> <span class="token punctuation">}</span> <span class="token keyword">private</span> <span class="token keyword">function</span> <span class="token function">retrieveMacroActivities</span><span class="token punctuation">(</span><span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token variable">$data</span> <span class="token operator">=</span> <span class="token variable">$this</span><span class="token operator">-></span><span class="token function">fetchMacroEstimatedData</span><span class="token punctuation">(</span><span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token variable">$this</span><span class="token operator">-></span><span class="token function">processMacroEstimatedData</span><span class="token punctuation">(</span><span class="token variable">$data</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token keyword">private</span> <span class="token keyword">function</span> <span class="token function">fetchMacroEstimatedData</span><span class="token punctuation">(</span><span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token comment">/* Fetch data */</span> <span class="token punctuation">}</span> <span class="token keyword">private</span> <span class="token keyword">function</span> <span class="token function">processMacroEstimatedData</span><span class="token punctuation">(</span><span class="token keyword type-hint">array</span> <span class="token variable">$data</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token comment">/* Process raw macro estimated data */</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span></code></pre></div> <p>The outcome is a service that is hard to follow and maintain. It is also more complex to test the different ways of retrieving the activities.</p> <h3>The Chain of Responsibility approach</h3> <p>The Chain of Responsibility pattern consists of having multiple services (called handlers) and running through them in a determined order to handle a request. <strong>Each service decides in runtime to either handle the request or pass it to the next handler</strong>. The CoR pattern can be used for instance in multi-stage validation or in managing multiple data providers.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/84e8d51d2705c6fa20558ed9e38bdab1/50383/CoR-schema.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 37.2972972972973%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAHCAIAAACHqfpvAAAACXBIWXMAABibAAAYmwFJdYOUAAAAwUlEQVQY031Qiw6DIAz0/39yy5JtieAEBcqjwE5R45JtF5rAtXe07WqtpRTOC0KIOMycEgfcYsqlJDwWJm1MzlpPMUYIOwQS1jpkhtco5UDkjbFCSqW094GIIJuN6XsBmXXucr3padrEMHaOcEEdWCLnvUf1PJvGA/hQaW0arIXdJj6A5mGMRjBIWAHfI9taODOfYmZU7KW+2WPsjCi1F8Lt2S/i1h6vwJawBWxoiZRgen88MfM/8S/g63FU2MWZfANKBZrcpPfZqAAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Schema of the Chain of Responsibilities pattern" title="Schema of the Chain of Responsibilities pattern" src="/static/84e8d51d2705c6fa20558ed9e38bdab1/50383/CoR-schema.png" srcset="/static/84e8d51d2705c6fa20558ed9e38bdab1/1d79a/CoR-schema.png 185w, /static/84e8d51d2705c6fa20558ed9e38bdab1/1efb2/CoR-schema.png 370w, /static/84e8d51d2705c6fa20558ed9e38bdab1/50383/CoR-schema.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>In our case, each data provider (meter reading, household consumption estimate, macro estimate) represents a "handler". A request is handled if the service returns an array of activities.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/51d2bab54be1ccb0c9f12fae822163d5/50383/activity-providers-using-cor.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 49.18918918918919%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAIAAAA7N+mxAAAACXBIWXMAABibAAAYmwFJdYOUAAAA/UlEQVQoz5WR626DMAyF+/7POERXrW0WCOQCxLl2J3jaqmlM2vmB4pjP9nFOj12llJQSDrVWIvKeQogIcemb2k0Lc+ZsjC17YjjGlGIKIbhlebveXrpeygGXo5rOr5eu6+9C5Jynee7PF2SVmr5h1LPOoT+6aGO0Ns4t67rhY4zVxnKIcYgC/vnROSIJGOWtdXIYPZFzbl1XY+2gFEbVWsPUbjAz9Qk/Cz4xP8qFvTwMUWBbcd9Ozbn8BRtjvkJwGIF3yeHm/SHcnBPxgV8BGwGZ94GFkNfbnZ/mFxiC8/SkwIoRawM5jGrbtkP4SLXUaZrFu8RC/g2zEd45wg/aIUmuVNL/1AAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Implementing activity providers using the Chain of Responsibilities pattern" title="Implementing activity providers using the Chain of Responsibilities pattern" src="/static/51d2bab54be1ccb0c9f12fae822163d5/50383/activity-providers-using-cor.png" srcset="/static/51d2bab54be1ccb0c9f12fae822163d5/1d79a/activity-providers-using-cor.png 185w, /static/51d2bab54be1ccb0c9f12fae822163d5/1efb2/activity-providers-using-cor.png 370w, /static/51d2bab54be1ccb0c9f12fae822163d5/50383/activity-providers-using-cor.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>To promote loose coupling, the 3 data providers will implement an interface that exposes a <code>retrieveActivities</code> function.</p> <div class="gatsby-highlight" data-language="php"><pre class="language-php"><code class="language-php"><span class="token keyword">namespace</span> <span class="token package">Component<span class="token punctuation">\</span>Activity<span class="token punctuation">\</span>Provider</span><span class="token punctuation">;</span> <span class="token keyword">interface</span> <span class="token class-name">ChainedActivityProviderInterface</span> <span class="token punctuation">{</span> <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">retrieveActivities</span><span class="token punctuation">(</span><span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></code></pre></div> <p>Each of the three services encapsulates the fetching and processing logic that match their respective data sources.</p> <div class="gatsby-highlight" data-language="php"><pre class="language-php"><code class="language-php"><span class="token keyword">namespace</span> <span class="token package">Component<span class="token punctuation">\</span>Activity<span class="token punctuation">\</span>Provider</span><span class="token punctuation">;</span> <span class="token keyword">class</span> <span class="token class-name">MeterReadingActivityProvider</span> <span class="token keyword">implements</span> <span class="token class-name">ChainedActivityProviderInterface</span> <span class="token punctuation">{</span> <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">retrieveActivities</span><span class="token punctuation">(</span><span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span><span class="token punctuation">:</span> <span class="token keyword return-type">array</span> <span class="token punctuation">{</span> <span class="token variable">$data</span> <span class="token operator">=</span> <span class="token variable">$this</span><span class="token operator">-></span><span class="token function">fetchData</span><span class="token punctuation">(</span><span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token variable">$this</span><span class="token operator">-></span><span class="token function">processData</span><span class="token punctuation">(</span><span class="token variable">$data</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token keyword">private</span> <span class="token keyword">function</span> <span class="token function">fetchData</span><span class="token punctuation">(</span><span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token comment">/* Process Data */</span> <span class="token punctuation">}</span> <span class="token keyword">private</span> <span class="token keyword">function</span> <span class="token function">processData</span><span class="token punctuation">(</span><span class="token keyword type-hint">array</span> <span class="token variable">$data</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token comment">/* Process Data */</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span></code></pre></div> <div class="gatsby-highlight" data-language="php"><pre class="language-php"><code class="language-php"><span class="token keyword">namespace</span> <span class="token package">Component<span class="token punctuation">\</span>Activity<span class="token punctuation">\</span>Provider</span><span class="token punctuation">;</span> <span class="token keyword">class</span> <span class="token class-name">HouseholdEstimateActivityProvider</span> <span class="token keyword">implements</span> <span class="token class-name">ChainedActivityProviderInterface</span> <span class="token punctuation">{</span> <span class="token comment">// Same structure as MeterReadingActivityProvider</span> <span class="token punctuation">}</span></code></pre></div> <div class="gatsby-highlight" data-language="php"><pre class="language-php"><code class="language-php"><span class="token keyword">namespace</span> <span class="token package">Component<span class="token punctuation">\</span>Activity<span class="token punctuation">\</span>Provider</span><span class="token punctuation">;</span> <span class="token keyword">class</span> <span class="token class-name">MacroEstimatedElectricityActivityProvider</span> <span class="token keyword">implements</span> <span class="token class-name">ChainedActivityProviderInterface</span> <span class="token punctuation">{</span> <span class="token comment">// Same structure as MeterReadingActivityProvider</span> <span class="token punctuation">}</span></code></pre></div> <p>Lastly, to link the services, we can use a “main” activity provider in which they are injected and ran through in the injected order. As soon as one of the handlers returns a valid response (a non-empty array in this case), we return its value. Note that the main activity provider handles the case where none of the chained providers returns a valid response.</p> <div class="gatsby-highlight" data-language="php"><pre class="language-php"><code class="language-php"><span class="token keyword">namespace</span> <span class="token package">Component<span class="token punctuation">\</span>Activity<span class="token punctuation">\</span>Provider</span><span class="token punctuation">;</span> <span class="token keyword">class</span> <span class="token class-name">ActivityProvider</span> <span class="token punctuation">{</span> <span class="token comment">/** * @var ChainedActivityProviderInterface[] */</span> <span class="token keyword">private</span> <span class="token keyword type-declaration">array</span> <span class="token variable">$chainedActivityProviders</span><span class="token punctuation">;</span> <span class="token comment">/** * @param ChainedActivityProviderInterface[] $chainedActivityProviders */</span> <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">__construct</span><span class="token punctuation">(</span><span class="token keyword type-hint">array</span> <span class="token variable">$chainedActivityProviders</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token variable">$this</span><span class="token operator">-></span><span class="token property">chainedActivityProviders</span> <span class="token operator">=</span> <span class="token variable">$chainedActivityProviders</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token comment">/** * @return Activity[] */</span> <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">retrieveActivities</span><span class="token punctuation">(</span><span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token class-name class-name-fully-qualified type-declaration"><span class="token punctuation">\</span>DateTime</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span><span class="token punctuation">:</span> <span class="token keyword return-type">array</span> <span class="token punctuation">{</span> <span class="token keyword">foreach</span> <span class="token punctuation">(</span><span class="token variable">$this</span><span class="token operator">-></span><span class="token property">chainedActivityProviders</span> <span class="token keyword">as</span> <span class="token variable">$chainedActivityProvider</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token variable">$activities</span> <span class="token operator">=</span> <span class="token variable">$chainedActivityProvider</span><span class="token operator">-></span><span class="token function">retrieveActivities</span><span class="token punctuation">(</span><span class="token variable">$startDate</span><span class="token punctuation">,</span> <span class="token variable">$endDate</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token keyword">empty</span><span class="token punctuation">(</span><span class="token variable">$activities</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token variable">$activities</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span> <span class="token keyword">return</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span></code></pre></div> <p>Finally, we need to inject the chained activity providers. In Symfony, one neat way of doing it is using the serices configuration file. This makes reordering as well as adding and removing the services an easy task.</p> <div class="gatsby-highlight" data-language="yaml"><pre class="language-yaml"><code class="language-yaml"><span class="token comment"># config/services.yaml</span> <span class="token key atrule">services</span><span class="token punctuation">:</span> <span class="token comment"># ...</span> <span class="token key atrule">Component\Activity\Provider\ActivityProvider</span><span class="token punctuation">:</span> <span class="token key atrule">arguments</span><span class="token punctuation">:</span> <span class="token key atrule">$chainedActivityProviders</span><span class="token punctuation">:</span> <span class="token punctuation">[</span> <span class="token string">'@Component\Activity\Provider\MeterReadingActivityProvider'</span><span class="token punctuation">,</span> <span class="token string">'@Component\Activity\Provider\HouseholdEstimateActivityProvider'</span><span class="token punctuation">,</span> <span class="token string">'@Component\Activity\Provider\MacroEstimatedElectricityActivityProvider'</span> <span class="token punctuation">]</span></code></pre></div> <p>With this setup in place, we call the <code>retrieveActivities</code> function of <code>Component\Activity\Provider\ActivityProvider</code> to retrieve a user's activities.</p> <h2>The trade-offs of the CoR pattern</h2> <p>Implementing the Chain of Responsibility pattern offered many advantages:</p> <ul> <li>Testing the main activity provider: We only need to <strong>test the behavior of the chain</strong>. Given a mocked chain of activity providers, the function should return the first valid result.</li> <li>Implementing the open/closed principle: the principle states that the behavior of classes should be <strong>changed through extension and not modification</strong>. This comes into play when we add, remove, or reorder the activity providers. We can easily change the behavior of the main activity provider by changing the configuration file.</li> </ul> <p>However, the usage of the CoR pattern comes with a few drawbacks, namely:</p> <ul> <li>Being unaware that the CoR pattern is implemented makes the <strong>code harder to understand</strong> and debug</li> <li>Depending on the used implementation, it is possible to have no handler to fall back to (if none of the defined handlers can handle the request). <strong>Always check for the fallback strategy</strong>.</li> </ul> <h2>Conclusion</h2> <p>In the long run, the Chain of Responsibility pattern helps organize and write modular code. However, it also comes with an initial cost higher than the trivial solution. They become interesting when the implemented system is prone to expansion. Indeed, setting up the chain is the most expensive part, while adding new chained elements keeps a constant cost over time.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 600px; " > <a class="gatsby-resp-image-link" href="/static/ea6dac096b8d1d3d3a4c2fe8a4da409f/0a47e/cost-per-change.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 64.86486486486486%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsSAAALEgHS3X78AAABZElEQVQ4y6WT626DMAyF+/5PuEr70xVa7lTccnVO7QAdoE7atEhHufqLYzsncAshYO3/I2knLG1d+Hvb2522dCJ6yXv/41zGohAIkwPyZkDb1HsPV+BquBodga91Pu+dgx8GFEmCj/MZ1trZQ8cbR6CsrcYimcd9Phf36gI2/QKmAeOjxZWhEWiMQVEUEfgNWIAi8ZAjIiByFr6p4JIL9yVo8TQstuJcBNZ1/QIG8kuol2CTQ2AvfHEDZQmorRjk5icL0Pt9UgRYlmWcaEfIHhZZ0SNLK5SfF+SsMU35WR2U8TEJyrgZFOb4bksuxtAvt5DRmNIE4/WK8XaH6nr0o0HezxflrUbWKNxZWRxraOt3ZbfPsiTHmjfVRdGbte6kXCzHWWAS7x3wWIeaQ6CVmqU1JCSGsxd7llI6yvBrLIvojYfbr7ctZpFi8MC1Nk1TvOC4v7X91dcTWFVVUV3Xvf98G9snsI/+o+JkrCYAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Comparison of cost per change Trivial solution vs CoR" title="Comparison of cost per change Trivial solution vs CoR" src="/static/ea6dac096b8d1d3d3a4c2fe8a4da409f/0a47e/cost-per-change.png" srcset="/static/ea6dac096b8d1d3d3a4c2fe8a4da409f/1d79a/cost-per-change.png 185w, /static/ea6dac096b8d1d3d3a4c2fe8a4da409f/1efb2/cost-per-change.png 370w, /static/ea6dac096b8d1d3d3a4c2fe8a4da409f/0a47e/cost-per-change.png 600w" sizes="(max-width: 600px) 100vw, 600px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>The implementation using injections is one of many ways to utilize this pattern. For implementations in other languages, refer to <a href="https://refactoring.guru/design-patterns/chain-of-responsibility">refactoring-guru</a>.</p> <p>Generally, design patterns should be perceived as blueprints that offer ready-to-use solutions to common problems. Since the human brain is built to work with patterns, understanding these solutions makes the code base more intelligible and easier to work with. However, one should be aware of the layer of complexity they can add due to their initial cost.</p><![CDATA[How to externalize large libraries from your application bundle?]]>/2022/04/externalize-large-libraries-outside-bundle//2022/04/externalize-large-libraries-outside-bundle/Wed, 27 Apr 2022 00:00:00 GMT<p>During the past months, I spent a lot of time trying to improve the frontend developer experience of about 15 developers on a large e-commerce website. The objectives included the reduction of our bundle size and CI run times. During this time, the development team introduced 3D assets, using the <a href="https://www.babylonjs.com/">BabylonJS</a> library to provide users with a more immersive experience regarding the articles displayed on the website. Unsurprisingly, this new feature increased our bundle size by 8 MB, representing more than 40% of our application size of the time. This in turn significantly increased our build times locally but also on the CI runners.</p> <p>To mitigate these consequences, the approved solution was to serve the Babylonjs library from a dedicated CDN excluding it from the bundle for each build.</p> <p>I am writing this article to share my learnings on how we managed to keep this dependency and more generally any large app dependency while reducing the consequences on the build size and time to a minimum.</p> <hr> <h2>TL;DR</h2> <ul> <li>Adding a large library increases bundle size and building times locally and on the CI/CD environments.</li> <li>Many solutions as lazy solution exist to separate large dependencies for client, but it does not solves the issues for building processes.</li> <li>To shorten building process, we externalized a large library to remove it from my application bundle.</li> <li>To do so, the library has been externalized in a script, and dynamically loaded the library logic from it, without modifying any import from my application</li> <li>Finally I measured the performance gain through a minimal example that you can find at the end of the article.</li> </ul> <hr> <h2>What are the impact(s) of adding large libraries as dependencies?</h2> <p>As a general rule, the bigger the package is, the greater the impact will be on a project's developer experience.</p> <h4>Download time</h4> <p>First, big dependencies impact installation times. When a dependency is added to a project, the designated packages and all its sub-dependencies are added to the <code>node_modules</code> directory. The time to compute dependencies, download the packages and sometimes build them will increase proportionally to the size of the new package. Yet, fast Internet access and cache have made these delays short, and exporting a whole package as an external dependency would not worth it if it was only to shorten these download times.</p> <h4>Build size</h4> <p>Another concern could be the bundle size increase related to the added dependency. However, modern bundling techniques such as lazy loading, code splitting and tree shaking already solve this issue by bundling only necessary code and splitting it into chunks. Moreover, moving packages into CDN would hide the problem because the user browser would still require to fetch the library. It is just not counted in the bundle served by the server.</p> <h4>Build time</h4> <p>Finally, the biggest issue with large dependencies is the build time increase. Building the application happens very frequently in the development workflow: building locally in watch mode when adding or modifying code or remotely with CI/CD. Globally, all the library code will be parsed and built into the bundle, so the larger the package is, the longer it will take to be included in the bundle. Externalizing the dependency on a CDN allows developers to retain the package functionality without having to include it at every build step.</p> <p>For instance, on my project, the frontend was managed by Next.JS and we regularly analyzed our bundle size with <a href="https://www.npmjs.com/package/@next/bundle-analyzer">@next/bundle-analyzer</a>. Just after installing BabylonJS and developing the 3D feature, a new chunk appeared, weighting more than 8 MB, which represented around 40% of total bundle size while increasing our CI build times by 30 seconds.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/1631942a3374ab2dab225b466582e876/50383/bundle-with-babylonjs.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 49.18918918918919%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAYAAAC0VX7mAAAACXBIWXMAAAsSAAALEgHS3X78AAACtElEQVQozw2Re09SARjG+U59gNbWunyJbrPWH5XdNLUi26KtrLXZzVauUirIQimBuKihyC04cRNRBDzE7cDhAKVohv46fzzbu/fdnv2e59WUoo/J+/opBLUUgzdVacn7b5JbHKAUuk3G5yPiiBF1CkRdAin3GoFnFbwjCtahFqPdfxk6scutEx0e9HbQKAkdlVAfCWc3y7OXSKkSLOdIuy/TiN+ikoyRDa+TmI8Rd0WI25P43pVZeC8zOdzg0cVNBrs63Dj7D93AXzSNxB1a8WuszJ4nMNmFMH1Gnc8hCz0o0QGk1Ri5cJ5MME1qfkm9x5geyWIZK/L+eQ1dT5O+8/+43ruDVruDph67TV24SjXUQ2qmm4K/l3qkX931kHcfp5wMkxPyJN1xfk4FCX+JYtJnMU9IGA0KIy//oBva5op2jyuDoJFzdkqrFsorZpqildbqcxrJ+yphL0XfBdY8HoSvEaJWQY26wOJEAKtD5NtclQmrwujH3wy/2+LuaAfdq100bWWJthxhu5nktzhJLTJI7Wcfku8kv+z7yYUWibuXCX708uOzH/cHDzOhAqaFMuNOGcP3Pzz5ss0D0y4PTXtoNuqrbCkx2tIcG0WVMPMGOXKDmv8YafM+qssRsovruJ+5cD2y4TUGcK6UmYpKGMIKhtAmH0I7vA3sMe5TCbeqCygZI8211yiJe2qXF6kGTlOYOULGdphEYg63N4jN4mDabMPl8TOeKTCWktAvNzGk2thyHdYbu+Tbaoet4iyN9CuUpSEqvi7k8AWy9qNkrPup/LiKZ93GlDiPWZxlUnRhKQXQSyVel+q8EFvo89tYJWgpMqLJqBrmjLRWhtUnXKY4c1A1PYXoOETecQDJewx7boI3WSv6ta+Mpc18Ks7zVhEZbtR42tzkpbyDcwOUcllNZOE/5EeM/j9meSkAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Bundle analysis showing a large BabylonJS chunk, made with @next/bundle-analyzer" title="Bundle analysis showing a large BabylonJS chunk, made with @next/bundle-analyzer" src="/static/1631942a3374ab2dab225b466582e876/50383/bundle-with-babylonjs.png" srcset="/static/1631942a3374ab2dab225b466582e876/1d79a/bundle-with-babylonjs.png 185w, /static/1631942a3374ab2dab225b466582e876/1efb2/bundle-with-babylonjs.png 370w, /static/1631942a3374ab2dab225b466582e876/50383/bundle-with-babylonjs.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <hr> <h2>How to do it!</h2> <p>There are three main steps to externalize a large dependency into a CDN:</p> <ol> <li>Configure the application bundler to load a global variable as an import</li> <li>Find or build a script to globally scope the library logics</li> <li>Dynamic loading of component once the external script is loaded</li> </ol> <h3>Bundler configuration</h3> <p>The purpose of a bundler is to take all the dependencies of a project and to split them all in a little number of files containing the module logic in order to improve the loading speed of the built application. This article is only going to focus on Webpack because it is the most used currently, but the method used here can be used on other bundlers as well.</p> <p>In order to bundle modules, Webpack does multiple operations:</p> <ul> <li>It recursively resolves all module imports from the application entrypoint</li> <li>It packs the associated code in a JavaScript asset and divides it into multiple chunks</li> </ul> <p>To externalize a dependency, Webpack needs to know which dependency we are talking about so that it may resolve the dependency imports differently. One way of doing this is to map the webpack dependency imports to a global <code>window.myDependency</code> variable. This is the goal of the next part of the article.</p> <p>Webpack has to be configured to not pack the dependency to externalize and to link all its uses to <code>window.myDependency</code>. A single configuration option allows to do that: <code>externals</code> (<a href="https://webpack.js.org/configuration/externals/">link to Webpack documentation</a>). This parameter takes an array of <code>string</code> or function returning a <code>string</code>. During the building, when Webpack reads an import, it tests if the dependency location is included in the <code>external</code> array (or function callback is called). If the matching element is a string <code>externalElement</code>, Webpack will replace the dependency location by reading directly into the variable <code>global.externalElement</code>. If the matching element is a function, it will follow the same process with the returned value.</p> <p>Here we want to replace the Babylon import <code>@babylonjs/core</code> by <code>BABYLON</code> due to the way they will be exposed in the CDN's script (explained in the next part). The Webpack configuration looks like the following:</p> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">module<span class="token punctuation">.</span><span class="token function-variable function">exports</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">config<span class="token punctuation">,</span> <span class="token punctuation">{</span> isServer <span class="token operator">=</span> <span class="token boolean">false</span> <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span></span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> config<span class="token punctuation">.</span>externals <span class="token operator">=</span> config<span class="token punctuation">.</span>externals <span class="token operator">||</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">;</span> config<span class="token punctuation">.</span>externals<span class="token punctuation">.</span><span class="token function">push</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> request <span class="token punctuation">}</span><span class="token punctuation">,</span> callback</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>request<span class="token punctuation">.</span><span class="token function">match</span><span class="token punctuation">(</span><span class="token string">"^@babylonjs/core"</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token function">callback</span><span class="token punctuation">(</span><span class="token keyword">null</span><span class="token punctuation">,</span> <span class="token string">"BABYLON"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token function">callback</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> config<span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre></div> <p><img src="/44f90d3b4cfe65decbe1882cb88f93be/external-import-diagram.svg" alt="Diagram for import resolves when external option is used"></p> <h3>Find or create the new library script and export it into a CDN</h3> <p>A dependency can not be externalized directly: the associated script has to have some modification. Indeed, the classic NPM or Yarn packages contain modules which are not strictly speaking scripts. The main purpose of modules is to expose logic (function, constants, etc.) that will be imported from other files. As explained in the previous part, the variable specified by the Webpack <code>external</code> option has to be accessible in the global scope. The dependency script must therefore be set globally when running in the browser.</p> <p>Most of the time, dependencies already provide usable scripts from an already existing CDN. This was the case for BabylonJS for example which simplifies the process a lot. If your dependency does not already provide a CDN solution, you have to build the library yourself. The simplest way to do this is to follow <a href="https://webpack.js.org/configuration/output/#expose-via-object-assignment">this Webpack documentation</a>. That allows to assign to <code>window</code> or <code>global</code> variables the elements exposed by the library. The result script should be the equivalent of one used with CDN.</p> <p>Once the necessary script is build, it can be easily hosted on a CDN like <a href="https://unpkg.com/">UNPKG</a> or <a href="https://www.jsdelivr.com/">JS Delivr</a>. Some libraries made this task even easier by providing these scripts through their own CDN. BabylonJS was one of these. For the ones who built their own files you can host them on a private CDN.</p> <h3>Conditionnal loading for components using the external script</h3> <p>If you followed the previous steps, you should have a fetchable script exposing one variable containing all the library logic and a Webpack configuration bounding the imports of the library to the global variable. The last part is to inject the script in the code. One approach would be to inject the script globally in the application. This would create performance issues for parts of the app that do not require the dependency. The other solution is to load the script dynamically. In this case, the library logic is not available once the application is running which means the application needs to be notified before actually being able to use the dependency.</p> <p>There are multiple ways to load a script conditionally and attach a callback once loaded. The simplest is by manipulating directly the DOM: adding a script with the <code>src</code> pointing to the script URL and <code>onload</code> calling the callback function. In our case, we were working with <a href="https://nextjs.org/">Next.js</a>, and since v12, this wonderful library provides a <code>&#x3C;Script /></code> component (<a href="https://nextjs.org/docs/api-reference/next/script">documentation link</a>) doing all this work! I used it this method to display my 3D asset (contained in the <code>&#x3C;Asset3d /></code> component):</p> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token punctuation">{</span> useState <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"react"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> Script <span class="token keyword">from</span> <span class="token string">"next/script"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> dynamic <span class="token keyword">from</span> <span class="token string">"next/dynamic"</span><span class="token punctuation">;</span> <span class="token keyword">const</span> Asset3d <span class="token operator">=</span> <span class="token function">dynamic</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token keyword">import</span><span class="token punctuation">(</span><span class="token string">"./Asset3d"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> <span class="token function-variable function">Asset3dContainer</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> <span class="token punctuation">[</span>isBabylonScriptLoaded<span class="token punctuation">,</span> setIsBabylonScriptLoaded<span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token function">useState</span><span class="token punctuation">(</span><span class="token boolean">false</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token punctuation">(</span> <span class="token operator">&lt;</span>div className<span class="token operator">=</span><span class="token string">"asset-3d"</span><span class="token operator">></span> <span class="token operator">&lt;</span>Script id<span class="token operator">=</span><span class="token string">"babylonjs-core"</span> onLoad<span class="token operator">=</span><span class="token punctuation">{</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">setIsBabylonScriptLoaded</span><span class="token punctuation">(</span><span class="token boolean">true</span><span class="token punctuation">)</span><span class="token punctuation">}</span> src<span class="token operator">=</span><span class="token string">"https://unpkg.com/babylonjs@4.2.0/babylon.js"</span> <span class="token operator">/</span><span class="token operator">></span> <span class="token punctuation">{</span>isBabylonScriptLoaded <span class="token operator">&amp;&amp;</span> <span class="token operator">&lt;</span>Asset3d <span class="token operator">/</span><span class="token operator">></span><span class="token punctuation">}</span> <span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token keyword">default</span> Asset3dContainer<span class="token punctuation">;</span></code></pre></div> <p>Multiple elements have to be taken in account:</p> <ul> <li>In my component, BabylonJS requires the external script to be fetch to have all the logic loaded. I used a state updated through <code>&#x3C;Script /></code>'s <code>onLoad</code> callback</li> <li>Once the script is loaded, BabylonJS logic is available in <code>window.BABYLON</code> so <code>&#x3C;Asset3d /></code> can be dynamically loaded</li> </ul> <p>After these 3 steps, my 3D asset was finally displayed on my page! A check shows that BabylonJS chunk disappeared from my bundle!</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/59f32294928290316a954a65b4abf660/50383/bundle-without-babylonjs.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 57.2972972972973%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAAsTAAALEwEAmpwYAAAC+0lEQVQozw3R3W9SBxjHcf6fXS5eLNnd4tWutngx5xa3aJbMOpIuszFdumbaaaZb7XTogLGqrfZFSxFbXkTmAUrLoabQUqTlxQOnwDmH1xb6Nlr87lx+kyefPMnPUAtfphq6RMH3Hen5C8j+Pr17aSyZ0KJpyuKa3jnKrgLSeJ7V0TLBqyVejjR4aGwyee2AWz1HmIc6XDV2MOxt+WknzWzHhqlHBqnFzdSiV6iJNtSFOEpoGVXcoLxW5I1DYnmsgMck47itMvpLA8vQLn9e07HLR3zfo4PNVRPVwEW0QC+K7zyVpZ9RvF8g+/pJCSGSfi+ymGBLlIk5syw6CryYKjL1t4JlpMaNoR0GfjzkovGYc98e62DyAdvrNuriFRTPGTT/N5QcJ6mEBmkUVNR4jFJ8k3XfOguuFIEXeVzPikxOlPnLWuHmH9tcH9ln4GaXngEd3JPmacV+pbkyjCYYUdyfodg/pLo4QGZdJCW6ySRiiOFVAsEUQjiPW9A/dCk8mqtzd2qHIds+/eZjem+/w6D5zpK3f0zZ+yWa5xTVl19RePIRkj5SpVlE0ZJIxRSJbJpQIoM7msMVLTK7pOEQmzwKtrnnPWR4rstvz3SwFr9HSbxFLdyHPHVCh8/y1v4psdmvWZNfE0nNE5NEEnIaX2YDZyqHN60yGVNxpXZwrO9ji3a4G+5yJ6iDu+UIrdQo9cUfkCZOkHWeJu+5QE7oZUWNImTtCJIfXy6CPRfjST7LjKwyka3ikNrM5A54njvCI4NbAkM7+ZhG4BKa8xNS5vdIWD9gw/y+PoKR51uzTKetzMoOnsoC01uvmShtMl1RuK82mFZ39f6Pt+0OlY03aNkMhkTrJ5ZLPYSlcwjpM7xKf85C4RSL9UH9eBxz4XcelMaYUQQeaxEs2hrWuoS11WC8dcDD7WOkvQ75sX+QFoIY1rojrLy7jkgfge55Augwp/m3089czcmoZsKs2HhaETBpr7DUY1haEjd2NUyHB9w5hM32PlVVpaxp/A8MVsyQSV7+bAAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Bundle analysis without previous BabylonJS chunk, made with @next/bundle-analyzer" title="Bundle analysis without previous BabylonJS chunk, made with @next/bundle-analyzer" src="/static/59f32294928290316a954a65b4abf660/50383/bundle-without-babylonjs.png" srcset="/static/59f32294928290316a954a65b4abf660/1d79a/bundle-without-babylonjs.png 185w, /static/59f32294928290316a954a65b4abf660/1efb2/bundle-without-babylonjs.png 370w, /static/59f32294928290316a954a65b4abf660/50383/bundle-without-babylonjs.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>With a wrapper as the one I showed above for your library, you should be able to display your component with all the library logic integrated while the latter is not considered as a dependency! Congratulations!</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 189px; " > <a class="gatsby-resp-image-link" href="/static/922efde990695c03b4eac1e7ab908d2c/0f8c7/3d-asset-radio.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 77.29729729729729%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAPCAYAAADkmO9VAAAACXBIWXMAAAsSAAALEgHS3X78AAADE0lEQVQ4y2WUXUhTYRzGx9yHuc+zj9xw5mblRvjNTGaQQsrQBNEkNqfDDxJElBZTkanI8gu8cZBkgnRjSEQMktVVECgJeiF4ISgYjMnmlbqLiYbz6ZxXz5z5wsvL4Zz39z7/5/+8hwN6XFxcMAvi8ThZd3d3MT8/j9nZWUxPT2NkZITMhYUFhEIhhMMRsjLj/PwcyQzO/7D9/X2Mj49jYGAA3d3dqKmpgdlshsFggC4zE4VFxRibmMCnpSUcHR9fQq/2MiwCZKGxWAy9vb0wmozIz88Dh8NJTIFQCB6fn3gWCfl4abMhdnJyQxABsrJXVlZQXV2N1FQhdDodJBIx0tPvQi6XQ61SQpp2B0pTAaiCMlAPc8GlwW9crxOl31LI+ObxeFBSUkJUiMViqNVqUDSQUiig8nyE1B+G+EsQ0q8hpLvfQSaVIBAIJKAJDw8PD4l3E7Q/w8PDyMrKQkoKFzIaptNqoNI/gORHFIKvYaT6fkH4YR1pP/9CXl6HOuuzaw/Z2nd2dmC32zE5OQmfz4f+/n5otVrwad8YqEjAg6rZDdn3I6QFjiD2R0B9/gOluRyGDC3CBweXHrLASCSC+oYG4uHMzAwWFxfhcrmgVCrB5XIhEokgEfIgMZhAFT6B9P4jKDLugaJLZg5nY5Qo+ezsjLyw2exob2+H1+sluRsaGoLJaCRKKYUSleVPIaHVst3WaDQYHR3F9vb27S6/n5uDvakJrzo7kZ2djaqqKkxNTZGA9/T0oLW1Fd63Y6irr8fz2lo4mpuJ3w6HA1tbW5fA5AwFg0G8aGxEi9OJWnqDQCAgJVdUVKCrq4uo7evrg9vtRltbG/E8JycHVqv1ZrCTr9C35WVU0sps9MdFxcWkVKbbAgHdHJmMZFMqlUKv14PHS0Fp6WNsbm5ex4YlM5NV6vf7SYNy83KJT0zQOzo6iMcMuKzMAqezhajb2Fi/IYiTfLGTQ763t0fKM5norlIUraSUlE5RclgsFgwODiYaEU++y7gayVD2NGacnp6Sv8/a2m+srq4SSDQaTbxP/pbZ+w/YTiFDbmNhdAAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="3D asset radio device" title="3D asset radio device" src="/static/922efde990695c03b4eac1e7ab908d2c/0f8c7/3d-asset-radio.png" srcset="/static/922efde990695c03b4eac1e7ab908d2c/1d79a/3d-asset-radio.png 185w, /static/922efde990695c03b4eac1e7ab908d2c/0f8c7/3d-asset-radio.png 189w" sizes="(max-width: 189px) 100vw, 189px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h4>Limits</h4> <p>Yet, not every dependency has to be moved into a CDN. There are three limits on externalizing a dependency from an application:</p> <ul> <li>Even if a CDN package is build, it has no other optimization as tree-shaking.</li> <li>Moreover, CDN are maybe fast, but they require an Internet access during development time.</li> <li>If your project is using TypeScript, then every import from the removed library in not typed. You may either ignore these issues locally but reduce type coverage, or you reinstall this dependency as a <code>devDependency</code>. In the second case, this would still mean downloading the dependency, but it won't be included in the final bundle or in the build process (except with Next.js, but <a href="https://nextjs.org/docs/basic-features/typescript#ignoring-typescript-errors">type checking can be disabled during the build</a> if already done previously).</li> </ul> <p>Thus, it is a cost-benefit trade-off between development time and use frequency of the package on one hand and building time on the other hand. In my case, the exported package was BabylonJS because 3D was only used on a few pages, but it also could be <a href="https://react-pdf.org/">React-pdf</a> if it is rarely used in the application, or any other large package not often used.</p> <hr> <h2>Conclusion</h2> <p>On my project, based on Github Actions data, I reduced the CI building time processes from around 30 seconds. To have better metrics, I created <a href="https://github.com/capsuleman/externalize-babylonjs">a minimal Next.js project</a> which externalize BabylonJS. In the commit history, the same 3D asset is displayed, first with BabylonJS as dependency, then without it. You can look at it to see a functional example.</p> <p>To test the performance of this change, I ran eight times a Next.js build command and timed it with the <code>time</code> command. Median building time decreases by 30 seconds, from <strong>64.4</strong> to <strong>33.295</strong> seconds! This time reduction is almost identical to the one without BabylonJS at all.</p> <p>Another benefit of shorter CI runs are lower costs. According to the current Github action pricing (0.008€ per second) and around 1000 builds per month, a reduction of 30 seconds on each build saves 240€ per month on the Github action bill.</p> <p>To conclude, externalizing a dependency is a complex process to setup, but it reduce all building process, by just removing the time used to build the dependency. It is worthless for small libraries, but useful for large ones, even more if they are used only on a little pages of your web application. I hope you enjoyed this article and use it to speed your CI/CD!</p><![CDATA[Working with a Third-party Provider, Three Lessons I Learned to Reduce the Lead Time]]>/2022/04/reduce-leadtime-provider//2022/04/reduce-leadtime-provider/Fri, 08 Apr 2022 00:00:00 GMT<p>For two years, I worked as a web developer with an investment bank. One of their key aims is to digitalize their finance system in order to speed up the procedure.</p> <p>At this moment <strong>the company was developing a more stable third-party API to provide the core data</strong> and which could manage the required performance for all of its consumers.</p> <p>Unfortunately, <strong>the new third-party service ran into a slew of issues</strong>. One of them was stability, and it's one of the most annoying, especially since the entire app depends on it.</p> <p><strong>The more reliant you are on a third-party service, the less control you have over your app.</strong> This suggests an <strong>increase in lead time</strong> when developing a feature or fixing an issue that is dependent on a third party.</p> <p>This page is about what the team tried and what we were successful at. But also where we failed to <strong>save time with our provider</strong> in order to deliver a defined quality to the consumer.</p> <h2>From how we started :</h2> <p>From the start of the project, we had a third-party owner in our team. His objective was to prepare :</p> <ul> <li>the migration of our core data,</li> <li>the connection to the third-party provider APIs,</li> <li>and ensure that everything worked flawlessly once completed.</li> </ul> <p>However, he left the project for a new adventure before we obtain the agreement for the migration. I volunteered to replace him. Everything was pretty well-prepared and documented, and I only needed to follow the process.</p> <p>But, the first weeks we plugged into it, <strong>we checked almost all the problems that you can ever have with a provider</strong> :</p> <p>✔️ <strong>Availability</strong></p> <p>✔️ <strong>Data integrity</strong></p> <p>✔️ <strong>Breaking change in business rules</strong></p> <p>✔️ <strong>Endpoint performance</strong></p> <p>Every day, we accumulated more and more problems, and we didn’t know when we will see the end. The time spent asking for investigation or fix increased drastically.</p> <p>From these issues, I learned how to shorten the time to make a request done and make issues disappear.</p> <hr> <h2>My First Lesson: Become aware of your problems before treating them</h2> <p>With experience, we reduce the investigation duration. However, <strong>switching context</strong> caused me to be <strong>less efficient in delivering the features</strong>.</p> <p>So I approached my IT lead and asked him, "What can I do?" His first answer was, “Do you realize how much time you spent on the third party problems?”</p> <p>Based on this question, I set a board where I <strong>record the request frequency, the time spent, the author, and the topic</strong>. The result was, in 20 days, I spent 255 min to treats 22 requests.</p> <p>This board also sheds light on two typical causes :</p> <ul> <li><strong>Poor communication</strong></li> <li><strong>Lack of autonomy</strong></li> </ul> <p>What I learned here is: that <strong>referencing all the out-of-context time, gives you a much clearer idea of how much time you spent on it, thinking about the issues, and will push to set up some actions.</strong></p> <p>So let’s tackle the causes now!</p> <hr> <h2>Second lesson: Making a request needs communication and negotiation</h2> <p>Knowing how to communicate the needs to the provider is a thing I learned and still learning in this project.</p> <p>Making an efficient request is not only about how clear you ask it. It's also about the <strong>information you provide</strong> to your interlocutor <strong>to convince them to prioritize your request</strong>. For that the <strong>information should be relevant</strong> to help him evaluate the work needed and <strong>to commit faster</strong>.</p> <h3>Identifying the interlocutor</h3> <p>To save time, asking for a request from the <strong>right person avoid doing some back and forth</strong> to relay information.</p> <p>So, identifying the positions of the provider team is the first thing I did.</p> <p>My next action was to define with the provider an owner for each case/field. Since what I learned was, that <strong>putting ownership in only one person will «empower»</strong> and lead him to put more effort to solve your request.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/f41cec9a63b37dd8c2c310a0899fab53/95fa1/contact_table.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 7.027027027027027%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAABCAYAAADeko4lAAAACXBIWXMAAAsSAAALEgHS3X78AAAAPUlEQVQI1z2JQQrAQAjE+hR1wYuedNX/v2wKQnsKSZ57L9wdzAwRQWYuieh3M4Oqrnc3Zmbb96sKEYFzDl6a9B3ygk7qPAAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Example of table where interlocutor is referenced." title="Example of table where interlocutor is referenced." src="/static/f41cec9a63b37dd8c2c310a0899fab53/50383/contact_table.png" srcset="/static/f41cec9a63b37dd8c2c310a0899fab53/1d79a/contact_table.png 185w, /static/f41cec9a63b37dd8c2c310a0899fab53/1efb2/contact_table.png 370w, /static/f41cec9a63b37dd8c2c310a0899fab53/50383/contact_table.png 740w, /static/f41cec9a63b37dd8c2c310a0899fab53/73caa/contact_table.png 1110w, /static/f41cec9a63b37dd8c2c310a0899fab53/536c7/contact_table.png 1480w, /static/f41cec9a63b37dd8c2c310a0899fab53/95fa1/contact_table.png 1816w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>An example was, I sent a request to the whole provider team to investigate why we couldn’t create any company. The answer I obtained was, “it’s not us it’s another team”.</p> <p>I then asked a developer who treats most of the time with them, the owner of the authentication process.</p> <p>The time to <strong>find the right owner took us 2 hours</strong> because he was not a part of the provider team. The fix, once we found the owner, took <strong>10 minutes to update the API key</strong>.</p> <h3>Know each other :</h3> <p>It happened pretty often that when it comes to explaining problems after investigations from our side or the provider side, it’s hard to understand because :</p> <ul> <li>we don’t know the architecture of each other systems,</li> <li>each team uses their abbreviations,</li> <li>we work differently.</li> </ul> <p>A good idea from one of our lead tech was to clarify all these points:</p> <ul> <li>Share the architecture of the application,</li> <li>Ask each time the definition of an abbreviation that we don’t understand, but also we use the less abbreviation possible, and only the ones that everyone knows.</li> <li>Ask how they do something if it takes longer than we expect.</li> </ul> <p><strong>Knowing their architecture, and how they work will make us understand the problematics, help us communicate more efficiently, and finally unlock situations faster.</strong></p> <h3>The importance of building a clear request :</h3> <p>What we learnt then was: <strong>it’s crucial to prepare your request</strong>.</p> <p>This mean that the request has :</p> <ul> <li>a <strong>clear why</strong></li> <li>accessible <strong>materials</strong></li> <li>a <strong>measurable</strong> goal</li> </ul> <p>With a <strong>clear request</strong>, you will make your provider <strong>understand better your needs</strong>, you’ll have <strong>less work to do</strong>, and it will be harder for them to refuse it facing facts.</p> <p><strong>A clear request is also a way to respect your provider</strong>. Advice from our sensei to make an even better request is to formally ask your provider to refuse the request if it’s not perfect.</p> <p>This is the best way to learn how to create the perfect interface for your specific relationship.</p> <p>It is essential that to <strong>respect the provider the same way you respect your customers</strong>.</p> <h3>The importance of the explaining the impact</h3> <p>What I often did before was in a hurry, ask them to investigate a problem, but without explaining to them the impact</p> <p>What could happen is that they don’t answer. They are busy and let you request again, to be sure that it’s important, or they would take their time.</p> <p>Here is a typical example I could do :</p> <blockquote> <p>– We have a problem, the private individual creation endpoint is on timeout, could you investigate on it?</p> </blockquote> <p>In this example, <strong>the provider doesn’t know the emergency level</strong> and will <strong>put the minimum effort to answer</strong> this.</p> <p>After several failures, my lead tech told me to learn the standard of the <a href="https://www.theodo.fr/time-to-market">company</a> to do a good request.</p> <p>The previous request became then :</p> <blockquote> <p>– We have an emergency problem right now, 10000 customers cannot create their account on the production environment, because the private individual creation endpoint doesn’t work since 14h. Could you investigate, please?</p> </blockquote> <p>Here you have a clear impact. It <strong>helps your provider to understand the emergency of your request</strong> and to prioritize it.</p> <h3>Make it faster : Bring the material</h3> <p>To reduce the time to convince our provider to help us, or to achieve a fix, I needed to <strong>prepare my talk and material</strong>. This <strong>help the provider to localize and take decision faster</strong>.</p> <p>There are plenty of <strong>tools</strong> that help us to gather all important information <strong>to prepare our requests</strong> to the provider. Here are some that we use :</p> <ul> <li> <p>Rest API client to test your provider APIs (<a href="https://www.postman.com/">Postman</a> / <a href="https://marketplace.visualstudio.com/items?itemName=humao.rest-client">RestClient</a>):</p> <ul> <li>What we do if their endpoint doesn’t work is to send them a screenshot of our Postman call, with the environment, URL, headers, body, response code, and response body visible. Your provider could extract more <strong>data from your screenshot</strong> than you imagine and can guide you.</li> </ul> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/86f0d61f210b2021ae1c3dcd640eb301/8bd7c/postman_2.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 51.35135135135135%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAYAAAC0VX7mAAAACXBIWXMAAAsSAAALEgHS3X78AAABkklEQVQoz3WS6W7bMBCE+f5P518pYLgGAqOSdfJaHhJFTpaU0xZJK+ADDw2H3EOEEFAxxoCCg50H2HVA5HkIEWQtiKhpYoxwzuE4DmQmpoynLZgo4/CE7CzE9XrFNE1QSvEhj2ANAmlsMWAcRzweD3Rdh77vm3E13bYN+743cqrwmvUVcbvdIKVswirYmjC1Q1rrdlHFe9/2vhIj87nms+JyuWAYht+mf4QR6yqbWfyH0f8Q9/u9mdXcbK8wPtFyhTMaqYbzV5hfdSfnf/H29gPLsjT3+pVS2phzhltn0DzicIY3DpyCl6Y08Wv9WjJC84GaH8cJ91uA52p6H5rhzpXcE1eUhTUFtdJ6I7g9nFoeHY8pehSzInNniKnvMM8zJBdg4RCVXGCU5HYxiMQvC3Q+jC+oHIUvKLlx5JPC8507xHJ6ROQXRL55Z9OgNJ4qYtQRk4kYeP5UAdIlpKOwGdgA4PZjQ3z7EkckiM3IEuj9HWoYcR8srr8Ubp3Gz16jXwmr8ZDWQxOb2xPy8Vthako+APzhB1ppvD5pAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Here the problem was because we were using an id token instead of an access token (screenshot from Postman)" title="Here the problem was because we were using an id token instead of an access token (screenshot from Postman)" src="/static/86f0d61f210b2021ae1c3dcd640eb301/50383/postman_2.png" srcset="/static/86f0d61f210b2021ae1c3dcd640eb301/1d79a/postman_2.png 185w, /static/86f0d61f210b2021ae1c3dcd640eb301/1efb2/postman_2.png 370w, /static/86f0d61f210b2021ae1c3dcd640eb301/50383/postman_2.png 740w, /static/86f0d61f210b2021ae1c3dcd640eb301/8bd7c/postman_2.png 845w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> </li> <li> <p>Monitoring tools (<a href="https://www.elastic.co/fr/kibana/">Kibana</a>, <a href="https://www.datadoghq.com/">Datadog</a>, ...):</p> <ul> <li>These tools allow us to visualize and <strong>analyze our application logs</strong> through different aspects like histograms, charts, ... We will see how essential it is below.</li> </ul> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/1c0f6cfe3ff436c683fb831c9d59b3e4/1c1a4/kibana_1.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 48.10810810810811%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAYAAAC0VX7mAAAACXBIWXMAAAsSAAALEgHS3X78AAAB5UlEQVQoz31S227TQBD1L/HIT/FRIMQDPPHEA1VVpagJQaVqEydNVDdxEt/vl7XX9vowYzeo4oEjj2Z3Z3zmqvl+CD8I4Xo+PBLX9VHVNRh93/9XGIp027ZQSiHNMmiWyPHB1EeCF2nJ2HQd2hfpOvXqPGq2M4lUHYqyRJ4XA7H22Vji3d0FvllPaOgBCvjLriiTrociQj6zrT+f+VM9/oX29bDB28uPePP9PS6cHTZpiC+HNT7tdQS1gF8JiKaBWaR4iD1ksoYrCsyCEyKyn8s+t0HzyxwTx8QP94Cf/gl3gY0rZ4/Lk4Gla+HePcGIAmyTAOvIg5GEWJGeeke4RTZW8jpDUQj0lUQnajRlBUkC2Q53x3JgH21EYQRJ95b8mlqiJh/2LYsSGfWO+5cXxdBLzaGpejTlMI4RRjEc1xt0nGbD9P1w3ALbGd/9IBp0QvYkTREnKSL2T5KBWOM0syzHfH6L6+sbhJQNgyfGPTmeLEync0wmN7Ao47btYNsuFssV9vsDttsnPD/vYZpHPD5uR0KOePv7fhDeRwavR9O0EKKCYeyw25moqhpSNkRsYzb7hSWRLhY6HkhWqw10fQ2tpp7wMnO6KZXhUGn8I68Qa0YcJ0NQBgcYAyqUpRjuUsohOOMPcqP2Zy0O3xcAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Here is a dashboard to monitor the response time of our provider route (screenshot from Kibana)" title="Here is a dashboard to monitor the response time of our provider route (screenshot from Kibana)" src="/static/1c0f6cfe3ff436c683fb831c9d59b3e4/50383/kibana_1.png" srcset="/static/1c0f6cfe3ff436c683fb831c9d59b3e4/1d79a/kibana_1.png 185w, /static/1c0f6cfe3ff436c683fb831c9d59b3e4/1efb2/kibana_1.png 370w, /static/1c0f6cfe3ff436c683fb831c9d59b3e4/50383/kibana_1.png 740w, /static/1c0f6cfe3ff436c683fb831c9d59b3e4/1c1a4/kibana_1.png 1046w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> </li> <li> <p>Sharing workspace for sharing documentation or joint action (<a href="https://www.notion.so/">Notion</a>:</p> <ul> <li>What we do with this tool is a <strong>sharing board</strong> where we <strong>track the current or future problems</strong> that could impact our provider or us. We put significant information that helps us to track the progress and to define the priority to treat them.</li> </ul> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/111714a5869d345314ab0bfe2d6cf6f5/b8c63/problems_table.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 18.91891891891892%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAECAYAAACOXx+WAAAACXBIWXMAABYlAAAWJQFJUiTwAAAAwElEQVQY0z2PwZKEIAxE/f8P3NMcZmpmxUCCCFWiK9hD0NpDVzqQPJoh54xSyr/WdcVoRnx+35hnDxbBmjfo3L7vOG51fxyotTaV3p/niYFogsq3xZQSSrtMy4LX84V9+4MNEYY9hBmsIovAgqXNMDuIXJ6IEEJQIMG0RFppuuB2IoibUUvtKb2XLhaGNwZJlxvEOQt2F9S0c62DmukG6UvblhHmhMfPE4tExBgbzPd0zhLehsAhId5AdwOttf3bX4gIMCTM+8ZpAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="problems table" title="problems table" src="/static/111714a5869d345314ab0bfe2d6cf6f5/50383/problems_table.png" srcset="/static/111714a5869d345314ab0bfe2d6cf6f5/1d79a/problems_table.png 185w, /static/111714a5869d345314ab0bfe2d6cf6f5/1efb2/problems_table.png 370w, /static/111714a5869d345314ab0bfe2d6cf6f5/50383/problems_table.png 740w, /static/111714a5869d345314ab0bfe2d6cf6f5/73caa/problems_table.png 1110w, /static/111714a5869d345314ab0bfe2d6cf6f5/536c7/problems_table.png 1480w, /static/111714a5869d345314ab0bfe2d6cf6f5/b8c63/problems_table.png 2712w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> </li> <li> <p>Draw schema (<a href="https://excalidraw.com/">Excalidraw</a>, <a href="https://whimsical.com/">whimsical</a>) :</p> <ul> <li>Explain problems, architecture or workflow with a drawing allow a better understanding. It makes <strong>easier to localize the problem, and to evaluate the time to fix it</strong>. Good advice that I learned was to <strong>draw with your customer/provider</strong> during a meeting help to <strong>synchronise your understanding</strong> of a problem.</li> </ul> </li> <li> <p>Alerting tools (<a href="https://www.sentry.io/">Sentry</a> + <a href="https://slack.com/">Slack</a>):</p> <ul> <li>To <strong>save time</strong>, and not be constantly checking our monitoring tools, we <strong>set a notification alert</strong> for each time something goes wrong from our provider server. This will help us to react faster, and to mention the problem to our provider. With this process, it happened that we <strong>detected the problem earlier than our provider</strong>.</li> </ul> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/2ce15c6c91fb33b1b77a542aa84cd2d8/dd104/sentry.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 43.24324324324324%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAYAAAAywQxIAAAACXBIWXMAAAsSAAALEgHS3X78AAABhElEQVQoz31S7U6DQBDkGYy2UuC4A67QUj5LqdBCo7FGE19BEx/AF/Jlx92zGmNSf2xub2d3ZpbDen99wSIrkRU1srxE3bRIswLrZkv3CkVZQwUavopgewpTV+KaznNh4eMNDzcdgu4eenxCPD4i3B0xp1O2B4hmhF32mFV72FLjmgin/xE+3+5wS+puvoFqBsi6NyGqHlE7Ih2OJr/QOa6IbOL4ZvCcU2ty5eAgA6y3Pbp+wHC4Q9N2ZsVAJ4jmC7hCwaHI6w2C+ZJIz69tKWpkwrrbI04LeMEcjtK4nImvsL9OW4SoSVQRId/Z6d9g15b0JAYhDYleZJDkKoiXCOPUuGGCMFkZjGt6mROeUi1FRDWlF2aGMUdGsHwiHH0Fl5yxw7RYmwaXQI9fN4zhkhg7n/khBN0F5Vzn8E4Y1xn/cRiSWrIqjeJMBPCjxAzalPMAD4oTOX8rrnPf9PRQv1ZW2Hs+krzGdjciySpD0PYjVtXGkC4JY7FV1ZgNvl2x6N9/8xNORQOyRpm+mwAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Example of a provider down notification" title="Example of a provider down notification" src="/static/2ce15c6c91fb33b1b77a542aa84cd2d8/50383/sentry.png" srcset="/static/2ce15c6c91fb33b1b77a542aa84cd2d8/1d79a/sentry.png 185w, /static/2ce15c6c91fb33b1b77a542aa84cd2d8/1efb2/sentry.png 370w, /static/2ce15c6c91fb33b1b77a542aa84cd2d8/50383/sentry.png 740w, /static/2ce15c6c91fb33b1b77a542aa84cd2d8/dd104/sentry.png 1064w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> </li> </ul> <h3>Establish good quality criteria/<a href="https://kpi.org/KPI-Basics">KPI</a></h3> <p>One day, the provider, announce to us that they are preparing for critical data migration.</p> <p>The risks if it is not done correctly can have impacts on :</p> <ul> <li><strong>data integrity</strong> ⇒ if you provide the wrong data, you lose the trust of the customer</li> <li><strong>performance</strong> of their SQL queries/ endpoint response time ⇒ <a href="https://developer.akamai.com/blog/2016/09/14/mobile-load-time-user-abandonment">a slow app (3 second to answer) can decrease 53% of your customer</a></li> <li>and the <strong>availability</strong> ⇒ If you have international customers, your apps need to be available 7/7 and 24/24, and you must inform your customer when there is any maintenance</li> </ul> <p>At this time, I was not confident to <strong>ask our provider for a guarantee</strong> that the process will work without any negative impacts.</p> <p>So I asked for help from our “sensei”. He made me realize that I was not confident because we didn’t have any good criteria to which the third party is committed.</p> <p>But <strong>what are good criteria?</strong></p> <p>Good criteria should be:</p> <ul> <li><strong>clear</strong> to make it easy to remember to constantly have it in mind</li> <li><strong>measurable</strong> to determine if it is reached, or if you will reach it soon</li> <li><strong>accessible</strong> with checkpoints to be able to check it at any time</li> <li><strong>realistic</strong> to be reachable</li> </ul> <p>To define our criteria, our first step was to <strong>collect all problems that we encountered and sort them to have a map of the actual situation</strong>.</p> <p>From this information, we could define a clear plan on what to measure.</p> <h2>Monitoring, monitoring, always monitoring ... and master your tools</h2> <p>In the beginning, we were monitoring the liveness of our provider endpoint by coding a scheduler that calls all the endpoints every 5 min and registers the endpoint state in our database. That was not a great idea.</p> <p>The result was <strong>poorly shareable</strong> for everyone, and we needed to layout the data to <strong>make it understandable</strong>, which took way too much time. We even deactivated it, because it created too many logs.</p> <p>Another bad idea was to monitor all the routes, by calling them several times to make an aggregate of the response time.</p> <p>We got the results, but it was <strong>manual</strong> and <strong>painful</strong> to do it frequently.</p> <p>We then asked our sensei for help again, and he made us realize, that we already have all the tools that we needed, but we didn’t know how to master them.</p> <p>To avoid all these pains and to save some time, we spent 2 hours studying our monitoring tools.</p> <p>We were then able to <strong>automatize</strong> the <strong>monitoring</strong> by defining the duration of a single event in our code (<a href="https://www.elastic.co/guide/en/apm/guide/current/data-model-spans.html">a span</a>), and using our monitoring tools (<a href="https://www.elastic.co/fr/kibana/">Kibana</a>) to display these events in a form of a dashboard.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/f66b5fa626aa46815555ecf4eb9bb90b/e8950/dashboard.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 44.86486486486486%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAYAAAAywQxIAAAACXBIWXMAAAsSAAALEgHS3X78AAACDklEQVQozz1Sy4oTQRTthX6B4MIfkdnoVtzoHwj+gTAL/QqdleLCB4IgM0MIyigiDjEjYRiDDsZo6GQy6Um60+l0+l3V1Y863qo4XrhQt/veU+ecW0YUxWB5DhX9bIkXgyGOFzkk1bKWKKsKcZIizwWqutZ9g2yFl+YJvtsMvBQQotDfs4zBUAMqbJ7i1q93uNvq4NXDA0hroUGrsoSUEufh5gy3e9TX+Ybnz3rwBxN4gY9CCOSUBuNcN7b8Ka53GziMXZxu7SLdP0ZYcLA0Q1XVKOWa3WE0x8bRNr6uZrj/+S0+DvtgcQxOKpUKI8syFHRIco57gzauHu1gc7CPcOUjDEOIokD9T6oKxfbB7zautV/jTv8TIln+/8d5DmMZBEiIhWDEhoZ/Rh4isfZUAalblc8WyTpNQthpjJMkwIf5GGbk61plWVZghGF0bQtd36FpiXOrRFXqBZR1hYI85NR442AXF5tbuLT3WOeV909xee8JLjQf4WanoedyWo4xjyP8WDqabhwn8KIIbpogJRAFpoAZY2g5EzRmJppTE29GPWyP+/q8Y/3BF9fSgGrBRkZynbOZlj2d2RgNx/BcD667gG07cOauvkhysaZf0VOKMzA/XNdZjvlkirPpbO2hYqCGatLrEpBpjrAKQiwWHsbjiQZW78wnrwNiX1K/8t0iAEEKCmLlUI/vr8hzib+QVJVrBTKYBwAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Kibana dashboard to measure the providers&#39; endpoints performance which lasts more than 2sec." title="Kibana dashboard to measure the providers&#39; endpoints performance which lasts more than 2sec." src="/static/f66b5fa626aa46815555ecf4eb9bb90b/50383/dashboard.png" srcset="/static/f66b5fa626aa46815555ecf4eb9bb90b/1d79a/dashboard.png 185w, /static/f66b5fa626aa46815555ecf4eb9bb90b/1efb2/dashboard.png 370w, /static/f66b5fa626aa46815555ecf4eb9bb90b/50383/dashboard.png 740w, /static/f66b5fa626aa46815555ecf4eb9bb90b/73caa/dashboard.png 1110w, /static/f66b5fa626aa46815555ecf4eb9bb90b/536c7/dashboard.png 1480w, /static/f66b5fa626aa46815555ecf4eb9bb90b/e8950/dashboard.png 2000w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>After getting the performance monitoring data in an <strong>accessible</strong> form, we came to our sponsor with some <strong>checkable</strong> numbers to tell them that we were not confident about the migration if they could guarantee us that 95% of their endpoints response time are below 2sec.</p> <p>This made them aware that they were not ready with some facts.</p> <p><strong>The impact was big</strong>. It results in our provider's PM coming to us to make an official communication that they had to delay the data migration, this in front of our sponsor.</p> <h3>Use monitoring to measure the <a href="https://kpi.org/KPI-Basics">KPI</a> and to negotiate with your provider</h3> <p>Here is another example of how monitoring helps us to <strong>convince our provider, with minimum time</strong>.</p> <p>Every time we encountered a data integrity problem we asked our provider to modify the data, but nothing will stop that if it happens again.</p> <p>We needed to find the root cause, and some long-term solution to prevent it from happening again.</p> <p>How? <strong>By gathering the material</strong>, we visually showed them:</p> <ul> <li><strong>how often does the problem occur</strong></li> <li>how much <strong>time both of us spent</strong></li> <li>how much <strong>time we could save</strong>.</li> </ul> <p>With this information, we reconciled our interests (not having bugs anymore) with their interests (not having to waste time on hotfix frequently) and <strong>proposed a “win-win” situation</strong>.</p> <p>We then <strong>convinced them to work together to find the root cause</strong> and fix it by adding a business rule in their code.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/880dc45bb4bc75510f569fcec07e5b37/89048/curiosity.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 68.64864864864865%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAAAsSAAALEgHS3X78AAADrUlEQVQ4yyWT2U+cVRjGv1rCMvvybcMynWGYYYeyI1C2dhj2YQQGhgGGYksRi0aaYqKmTaNGmihgI4JiE6i9aLwwsXrTRImxcGG9bLw1avTv+PlSL96c97w5ec6znKOMNjWyND5OemCA0Y5ORi50MdYnfd8g7Y0v098TZXYyRUttEyMXY8SjMZZSU2QGo6wkBllLjZEZ7mf98iw3EpdQVpKTHB2fcPLsGQ8ODnh4eMAff//F89+fc/ejDY5Pjjn65Sk7H7zLV/f3efLTj/z5z798+/0PPDn6meNff+Pg64fs7u5xf2sDJVEWYDnaxZXEK3R39NDb0sG1VJq5qVkSvT0kYgO82neBnalmVsf6mIgN8UZ6iiVhtzoZ5+ZskqX4EIs9DdyM1aCst3nZjjrYSjUx311Fm+kl6lfxnLUQcbvwWezMhRy8X+9muVJluiqfiZCHUb+D4UIHg6aNi4acCTtYO29BSZdqrNRqzNcWcilSQKohzMZUK+3hIHX5PsKGj5bCfBoMnZYCH1fl0tfbw8xVGqQrfCRLfYyGfCzVGXzYX4jSa2g0e1RKPV50u5eAw0uN203II3MBLJZ9ZzjEa531JBpqKLO6GApoLNboLFTqzEQ0xkMaGWH/Tk8BSpep0aprnHNquK1eIgJU6dUpcWsEZaZbPNT4fCw2VZOsr0DN8xBw6oyFdcbLTPpDBr0Bg3hE1vJ8lGZNo0nVqBbQclnPy77Vp1Pq0lAtKt48Ye30MFcfYSho4BMVjhyPsNfFDgNTldIMuoMmtQFhWOsVMGFUIUDV4lOx26DdbxKWmSNPxbSqLLQFuNEdlLBMTKeKI1cloplooiAr14NVzpwCthYLwyqR6Lf/758jx85ZJRdXjg3nS1aype/W7azX2chELDR7XVjO5KEoWeRL+kU2u/R5FAlGvMSgrshECcqNPptK1hkbjfIbkuNT9MuP6OuNkUrOcG0iwcpoP9fT06QGYkzEx0mMTTLQG2UmHmdkeOyF9C7xsb1EGJ56VeDQhFkOC+kMjx9/x/XlFTY//oQHh4dsb39GcnqW/f193lt/my/3vuCbR4+4c/s2n+/scOfWLTSrE7/XoEQ3TiXrGJKkXbzQJWV3rlOSdVLgkhRHE8zNzDM9Oc2VzGWuzmd4a/VNaiIVeHOsFDrk4ducBMXvkARaJUyVCtFvCpAl2y0JCriUmuui3B9md2+XtbU1Njc32dre5tN793h6fMxIdAAt24LfY7zw3pQypM65Vf4DYzXfxCS1B5sAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="curiosity" title="curiosity" src="/static/880dc45bb4bc75510f569fcec07e5b37/50383/curiosity.png" srcset="/static/880dc45bb4bc75510f569fcec07e5b37/1d79a/curiosity.png 185w, /static/880dc45bb4bc75510f569fcec07e5b37/1efb2/curiosity.png 370w, /static/880dc45bb4bc75510f569fcec07e5b37/50383/curiosity.png 740w, /static/880dc45bb4bc75510f569fcec07e5b37/73caa/curiosity.png 1110w, /static/880dc45bb4bc75510f569fcec07e5b37/89048/curiosity.png 1242w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p><strong>Using visual management helped us to have much more impact to negotiate with our provider</strong> to investigate the root cause instead of fixing the problem temporarily. We then never heard about the problem again until now.</p> <h3>Now commit!!</h3> <p>We saw how to do a great request, but still, something is missing.</p> <p>The last point is to <strong>make your provider commit</strong> to it. A good commitment, has a confirmation message, with a <strong>check date</strong> and an identified <strong>owner</strong>.</p> <p>So you know when to check and know who to talk with when the request is still not resolved.</p> <h2>Go further</h2> <h3>Go drink a beer ... or a tea with the provider team :</h3> <p>A thing that we didn’t achieve was to invite the provider team to a bar.</p> <p>This is not a joke. We could figure it out that they are probably not so different from you.</p> <p>This could also make us <strong>aware of their working environment</strong>, and how they work, and so <strong>better interact together</strong>.</p> <hr> <h2>Third lesson : Share your tools and methodologies</h2> <p>Another important lesson, that I learned during these years was how to avoid overwhelm when I was in front of tons of requests from others teams or our providers.</p> <p>What I learned was to delegate the request but not anyhow.</p> <p>I aimed to <strong>make everyone a copy of myself when it comes to the method and knowledge</strong> of my topic.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/8148b6d36ddd168be779a91a9b51c99f/69d6b/clone.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 56.21621621621622%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAAsSAAALEgHS3X78AAAC6ElEQVQozw2Ty28aVxSHp2lapXLkKgk4CrWCDQYbzGtgeBlmGIyBGTMY8IB5xC8c3AZoUtW22kaxs2idVChdJKkaqYtsuuuy6qp/3de7uqtzz/f9zjnSzYqOtFVAyiZZsaqEd6pIqRR3/EEcix4c7hUWwzEWTJPb9RqSsYVU1pB0lRuayq1EEkdcodjeo2rvIa01LA4uxqR2m0waFQpanlvioztuL/NrIbxLy+Q3NEqVHdKbBt6tChHbZq3d5nNVZT6mcC+aYCkaJ5DOIVmaxvnAZrxrUQx6CEUj+ALr+IMhHIJMicuEAgFcD1y4Hyzi9/pwByN4PD7uuv2s+AMsJISRdxV/LIF0oMapJSOs+nyUzRolwyAhR3Elszj1MgGzTnpjg3QkQDgSxplJcVuW8Xo8osaLZ/kha8JqbnGJOed9pLv3nHx530Uql+fb8QR7OMJh1LnR6yMdPkJq7PKpafGwqLOuxJkvFJhTFLyCum5W8AWDmCLXaDTGZwsuJKfIaa9Vo2MWKKkKbr3AzVodaXiA9OwEabKPdNpn+dAm1W0wn82KIcT5IiTzuN+k2TRJJGXUbJqVhFBWy2WedJsM9rs8HfYoF3NEjDLepoVj1MO8/AZ90kOvbzKwW6waFbZaFmqlxNvZBUejLp8sfMVGUhG1Yvo/Xkz5+P4lT22LRkllVZGFSonTky7FqsbV+WOeDdukTZWRqdPSMzQqm9jlItPJMW9++xUlkeGqb/Fu9hzpv3//4eOHP/hr9oq//3zP1ctzjnYsjkU+stD7+fU17cEhOTXF5WlbKBps5+K0cwnsVgdru8eBvsWgbnD5/HukwdEYOVPCSCu8evEdP/0wYV+QtbQUlXyOXc2gmtGoi6V/c33Gh3eXlESzViHLsKrzRNC+ftQkGorx4mwsjiSeoJnLEF6XmU6nTMZT2nqek2qOX7pVZv1tZuI961nstFp0Ox1BWyAVXqejZfj96w7XxzZpsYPD4xH/AzKHiABo/tkQAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="clone" title="clone" src="/static/8148b6d36ddd168be779a91a9b51c99f/50383/clone.png" srcset="/static/8148b6d36ddd168be779a91a9b51c99f/1d79a/clone.png 185w, /static/8148b6d36ddd168be779a91a9b51c99f/1efb2/clone.png 370w, /static/8148b6d36ddd168be779a91a9b51c99f/50383/clone.png 740w, /static/8148b6d36ddd168be779a91a9b51c99f/73caa/clone.png 1110w, /static/8148b6d36ddd168be779a91a9b51c99f/536c7/clone.png 1480w, /static/8148b6d36ddd168be779a91a9b51c99f/69d6b/clone.png 1912w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p><strong>Teaching to make everyone autonomous will release your stress, gain time, and last but not least, put your footprint on the methodology.</strong> So even if you leave, people will work as you do.</p> <p>In this paragraph, I will tell you how I made things work as I was present, and not to be overwhelmed with my problems.</p> <h2>Make your team autonomous</h2> <h3>Help your team to identify the problem</h3> <p>5 different teams, depending on us to get the provider data.</p> <p>In the beginning, we got contacted multiple times a day for data reading or writing problems.</p> <p>However, the provider data could be recomputed with many business rules set by others teams through different layers.</p> <p>To overcome this situation, we helped everyone to <strong>identify the origin of the problem</strong>. For that, we set a status error (424) with a <strong>message containing the problem</strong>, when the problem comes from our provider.</p> <p>Since that, <strong>the contact frequency reduced drastically</strong> and the teams became much more compliant.</p> <h3>Help your team pay attention to the provider problem</h3> <p>Previously I showed you that <strong>alerting tools</strong> can be <strong>great to gather material</strong> and be notified if something goes wrong.</p> <p>But there is also a <strong>drawback</strong> that we experienced by not mastering it.</p> <p>What we didn’t do well, was logging our error. The alerting tools caught all the error logs, which some of them were not, and sent <strong>too many false positive notifications</strong>.</p> <p>The consequence was that we became immune from it, and we didn’t pay attention to the notifications.</p> <p>So it is <strong>important to set up well your alerting tools and your log</strong> to make everyone use it.</p> <h3>Monitor your requests</h3> <p>In my first lesson, I learned to monitor all the requests, and the result was <strong>in 20 days I spent 255 min to threatening 22 requests</strong>.</p> <p>So what did I do then?</p> <p>I sorted all the requests and realized that most of them are easy to solve.</p> <p>What they required is to know in which endpoint they could find the specific information they needed.</p> <p>The problem was no one was familiar with these endpoints, and there is no easy way to get the information.</p> <p>To solve this, I <strong>made a Q and A</strong>, postman collection, and documentation relative to the most frequent requests, <strong>to make everyone autonomous</strong>.</p> <p><strong>The 255 minutes dropped to 10, with 2 requests.</strong> Finally, I had some time to drink a coffee!</p> <p>So <strong>teaching</strong> the team is time-consuming, but <strong>it’s a great investment in the long term</strong>.</p> <p>If you work 6 months on the project, I let you count how much time you gain, and what you can do with that?</p> <h2>Make your provider autonomous in front of problems</h2> <p>A typical problem that we encountered was the increasing response time each time they did a deployment, for different reasons.</p> <p>This leads us to always check the response time of all of their endpoints, after each deployment. It took us 30 min each time.</p> <p>To avoid this, <strong>we shared our monitoring response time tools</strong> with them with the endpoints that we used the most.</p> <p>It cost us 30 min to create an access login and to teach them how to use it.</p> <p>With that, we are pretty confident, that they will <strong>be more aware of the impact</strong> of their deployments and fix it if necessary.</p> <p>And this would make them also <strong>more reactive</strong> since they are committed to the response time that they need to deliver.</p> <p>The result is that our provider, check frequently the dashboard, and contact us by themselves to tell us that <strong>they improved the response times</strong> of some endpoints which were already less than we expected.</p> <h3>Don’t miss any requests : track it with a co-board:</h3> <p>When it comes to new version deployment or data migration, to follow critical request resolution, we created an exclusive channel during important events.</p> <p>But it would be <strong>counterproductive to create many channels for smaller requests</strong>, like data/logs analysis.</p> <p>What we tried was to implement a light <strong>Kanban board</strong> (we used <a href="https://www.notion.so/">Notion</a> for that) <strong>to track progress</strong>, and don’t forget any of them. This board could be filled by the problem owner team.</p> <p>It is also useful to track the weekly/monthly meeting with the provider if you couldn’t attempt it.</p> <p>To be honest, our <strong>first try doesn’t work</strong>. Our provider told us that they will have a hundred consumers and couldn’t maintain a board like that, if they have to do it for every one of them.</p> <p>What we are planning to do is to maintain it at the beginning by ourselves, to track the requests that we needed to ask again, and the time spent. This will <strong>give us some material</strong> to determine if we should convince harder our provider to use it.</p> <h3>Invest into long term by teaching</h3> <p>There are two ways to put your footprint on a project:</p> <ul> <li>you work a lot, it’s worked, hurra everyone will congrats you 5 min. However, you lost your hair and the problem should be fixed well, because the day you disappear, I let you imagine the rest of the story.</li> <li>you are pretty smart (or have a good sensei), you <strong>teach everyone little by little a routine about how you do to deal with the provider</strong>. The day you disappear, everything is still working as you were here. You succeed to <strong>put your footprint on this ownership</strong>. <strong>People will work as you teach them</strong>. So you can enjoy your vacation.</li> </ul> <hr> <h2>Conclusion</h2> <p>What I can retain from all of these experiences was, that there is <strong>always a way to make things done by becoming proactive</strong>.</p> <p>It’s not easy, and it’s time-consuming. But, you should not think about losing time, but <strong>investing time for the long term</strong>.</p> <p><strong>To sum up</strong> what I've learnt over this project:</p> <ul> <li><strong>Communicate clearly</strong> your request to the right interlocutor <strong>with a why, materials</strong> to only post it once with respect, and <strong>expect them a commitment</strong>.</li> <li><strong>Monitoring</strong> the most sensible “points” and <strong>making them accessible</strong>, would make you gain a bunch of time to prepare your request, and convince your provider to prioritize it.</li> <li><strong>Before looking for new tools or coding</strong> some new ones for monitoring, <strong>look at what you already have</strong>, and <strong>master it</strong>, it can be better than what you expect.</li> <li><strong>Make everyone autonomous to decentralize your task</strong>. This avoids you from switching contexts, and <strong>put your footprint on the methodology</strong>. The project will still work even if you disappear ... at the coffee shop.</li> </ul> <p>... And last but not least don’t forget the drink with your provider, it’s important to drink!</p><![CDATA[Serverless, DynamoDB, Streams, Filter Patterns and You]]>/2022/04/dynamodb-streams-serverless-filter-patterns-and-you//2022/04/dynamodb-streams-serverless-filter-patterns-and-you/Wed, 06 Apr 2022 00:00:00 GMT<p>This article will cover filter patterns in the serverless framework as they apply to DynamoDB streams, but the principles apply to filter patterns on other event types as well (kinesis, event bridge, etc...)</p> <hr> <p>DynamoDB Streams provide near real time information on all data modifications that occur in your database at no extra cost (No paying for event messages or any in between computations). We can then use these streams to trigger Lambdas without having to setup any extra infrastructure! Read more about DynamoDB Streams <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html">here</a>.</p> <p>This sounds great but quickly the problem becomes that you have a Lambda triggered on every single data modification for your DynamoDB table wasting the saved costs by spinning up a Lambda whose only job is to recognize it has no job to do and then stop processing. The answer to this problem is to set up filter patterns. Filter patterns allow you to execute a listener to an event only under certain circumstances removing the data validation part of our Lambda and rely on our filter pattern to only invoke the Lambda when necessary. Plus stream filter patterns come with no extra costs making them the perfect solution.</p> <hr> <h2>Before we dive in</h2> <p>Filter patterns work on any event structure but our examples are specific to DynamoDB streams which have an event structure like the following:</p> <div class="gatsby-highlight" data-language="json"><pre class="language-json"><code class="language-json"><span class="token punctuation">{</span> <span class="token property">"eventName"</span><span class="token operator">:</span> <span class="token string">"INSERT"</span> | <span class="token string">"MODIFY"</span> | <span class="token string">"REMOVE"</span><span class="token punctuation">,</span> <span class="token property">"dynamodb"</span><span class="token operator">:</span> <span class="token punctuation">{</span> <span class="token property">"Keys"</span><span class="token operator">:</span> <span class="token punctuation">{</span>...<span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token property">"NewImage"</span><span class="token operator">:</span> <span class="token punctuation">{</span>...<span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token property">"OldImage"</span><span class="token operator">:</span> <span class="token punctuation">{</span>...<span class="token punctuation">}</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span></code></pre></div> <p>The expamples below use the serverless-lift plugin to create our database instances. This is why the stream arn's look so nice they are a direct reference to a database created in the serverless file. Read more about serverless lift and DynamoDB <a href="https://github.com/getlift/lift/blob/master/docs/database-dynamodb-single-table.md">here</a>!</p> <hr> <h2>How to implement filter patterns in your project</h2> <p>Lets say you have a series of jobs making updates to a single record and you have a lambda that is going to send an email to your users when the jobs finish processing. To denote when a job is finished there is a jobState column on your record. You could have a serverless lambda function setup like this:</p> <div class="gatsby-highlight" data-language="yml"><pre class="language-yml"><code class="language-yml"><span class="token key atrule">emailUsersWhenJobComplete</span><span class="token punctuation">:</span> <span class="token key atrule">handler</span><span class="token punctuation">:</span> src/emailUsersWhenJobComplete.handler <span class="token key atrule">events</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">stream</span><span class="token punctuation">:</span> <span class="token key atrule">type</span><span class="token punctuation">:</span> dynamodb <span class="token key atrule">arn</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>construct<span class="token punctuation">:</span>batchJobs.tableStreamArn<span class="token punctuation">}</span> <span class="token key atrule">filterPatterns</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">eventName</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>INSERT<span class="token punctuation">,</span> MODIFY<span class="token punctuation">]</span> <span class="token key atrule">dynamodb</span><span class="token punctuation">:</span> <span class="token key atrule">NewImage</span><span class="token punctuation">:</span> <span class="token key atrule">jobState</span><span class="token punctuation">:</span> <span class="token key atrule">S</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>DONE<span class="token punctuation">]</span></code></pre></div> <p>Let's break the filter pattern down.</p> <p>First the filter pattern says to only pay attention to <code>INSERT</code> or <code>MODIFY</code> events meaning any remove events are automatically ignored. The next few lines of the filter pattern drill down on the event structure to get to the attribute that we need. Then it says that we expect the value of <code>jobState</code> to be <code>DONE</code>. Meaning any inserts or updates to our table records that happen before the job is done are ignored by our pattern and our lambda doesn't even spin up.</p> <p>This is good, but what about updates that happen to our record after it hits the done state. Those will trigger our lambda to run as well causing us to send emails we don't want to send. So we only want it to process when the <code>jobState</code> wasn't <code>DONE</code> before the modification. We can accomplish that with filter patterns as well. Our serverless lambda function will now look like this:</p> <div class="gatsby-highlight" data-language="yml"><pre class="language-yml"><code class="language-yml"><span class="token key atrule">emailUsersWhenJobComplete</span><span class="token punctuation">:</span> <span class="token key atrule">handler</span><span class="token punctuation">:</span> src/emailUsersWhenJobComplete.handler <span class="token key atrule">events</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">stream</span><span class="token punctuation">:</span> <span class="token key atrule">type</span><span class="token punctuation">:</span> dynamodb <span class="token key atrule">arn</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>construct<span class="token punctuation">:</span>batchJobs.tableStreamArn<span class="token punctuation">}</span> <span class="token key atrule">filterPatterns</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">eventName</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>INSERT<span class="token punctuation">]</span> <span class="token key atrule">dynamodb</span><span class="token punctuation">:</span> <span class="token key atrule">NewImage</span><span class="token punctuation">:</span> <span class="token key atrule">jobState</span><span class="token punctuation">:</span> <span class="token key atrule">S</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>DONE<span class="token punctuation">]</span> <span class="token punctuation">-</span> <span class="token key atrule">eventName</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>MODIFY<span class="token punctuation">]</span> <span class="token key atrule">dynamodb</span><span class="token punctuation">:</span> <span class="token key atrule">NewImage</span><span class="token punctuation">:</span> <span class="token key atrule">jobState</span><span class="token punctuation">:</span> <span class="token key atrule">S</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>DONE<span class="token punctuation">]</span> <span class="token key atrule">OldImage</span><span class="token punctuation">:</span> <span class="token key atrule">jobState</span><span class="token punctuation">:</span> <span class="token key atrule">S</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">anything-but</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>DONE<span class="token punctuation">]</span></code></pre></div> <p>This updates the filter pattern we had before and splits out the insert and the modify into seperate patterns. AWS applies filter patterns in a way that ensures the lambda will be triggered if the stream event matches any of the provided patterns.</p> <p>The <code>MODIFY</code> event has also been updated to reference <code>OldImage</code>. This is the state the database record was in before the modify event was emitted. By doing this, we have now specified that if an update happens and our record ends up with a <code>jobState</code> of <code>DONE</code> we still only want to trigger our lambda and send an email if the <code>jobState</code> before this update was not <code>DONE</code>, solving our earlier problem. Users will now only get notified when the job finishes and not if any updates are made after the job finishes.</p> <p>What else can filter patterns do? Quite a lot. Lets say we have a shopping platform and we want to send our users a coupon if they make an order over $100.00. We could have a serverless lambda setup like this:</p> <div class="gatsby-highlight" data-language="yml"><pre class="language-yml"><code class="language-yml"><span class="token key atrule">sendCoupon</span><span class="token punctuation">:</span> <span class="token key atrule">handler</span><span class="token punctuation">:</span> src/sendCoupon.handler <span class="token key atrule">events</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">stream</span><span class="token punctuation">:</span> <span class="token key atrule">type</span><span class="token punctuation">:</span> dynamodb <span class="token key atrule">arn</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>construct<span class="token punctuation">:</span>orders.tableStreamArn<span class="token punctuation">}</span> <span class="token key atrule">filterPatterns</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">eventName</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>INSERT<span class="token punctuation">]</span> <span class="token key atrule">dynamodb</span><span class="token punctuation">:</span> <span class="token key atrule">NewImage</span><span class="token punctuation">:</span> <span class="token key atrule">total_amount_cents</span><span class="token punctuation">:</span> <span class="token key atrule">N</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">numeric</span><span class="token punctuation">:</span> <span class="token punctuation">[</span> <span class="token punctuation">></span><span class="token punctuation">,</span> <span class="token number">10000</span><span class="token punctuation">]</span></code></pre></div> <p>Or if we need to run extra structural checks when a user adds a pipe with a radius greater than 5 cm, a length greater than 10 m, and any PVC based material type in a 3d modeling system:</p> <div class="gatsby-highlight" data-language="yml"><pre class="language-yml"><code class="language-yml"><span class="token key atrule">structureCheck</span><span class="token punctuation">:</span> <span class="token key atrule">handler</span><span class="token punctuation">:</span> src/structureCheck.handler <span class="token key atrule">events</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">stream</span><span class="token punctuation">:</span> <span class="token key atrule">type</span><span class="token punctuation">:</span> dynamodb <span class="token key atrule">arn</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>construct<span class="token punctuation">:</span>structures.tableStreamArn<span class="token punctuation">}</span> <span class="token key atrule">filterPatterns</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">eventName</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>INSERT<span class="token punctuation">]</span> <span class="token key atrule">dynamodb</span><span class="token punctuation">:</span> <span class="token key atrule">NewImage</span><span class="token punctuation">:</span> <span class="token key atrule">type</span><span class="token punctuation">:</span> <span class="token key atrule">S</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>piping<span class="token punctuation">]</span> <span class="token key atrule">radius_cm</span><span class="token punctuation">:</span> <span class="token key atrule">N</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">numeric</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token punctuation">></span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">]</span> <span class="token key atrule">length_m</span><span class="token punctuation">:</span> <span class="token key atrule">N</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">numeric</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token punctuation">></span><span class="token punctuation">,</span> <span class="token number">10</span><span class="token punctuation">]</span> <span class="token key atrule">material</span><span class="token punctuation">:</span> <span class="token key atrule">S</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">prefix</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>PVC<span class="token punctuation">]</span></code></pre></div> <p>As you can see pairing DynamoDB streams with serverless filter patterns lets you perform in-depth event logic without having to spin up any extra infrastructure or invoke any extra functions. Next time you need to publish an event after a database write in your serverless project why not try out a filter pattern. You just might save yourself some time and money.</p> <hr> <h2>Links</h2> <p><a href="https://unsplash.com/">Images from unsplash</a></p> <p><a href="https://docs.aws.amazon.com/lambda/latest/dg/with-ddb-example.html">Information on DynamoDB stream event structures and testing events</a></p> <p><a href="https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html">Full list of filter pattern rules and raw json filter pattern examples</a></p> <p><a href="https://github.com/getlift/lift">Information on the Serverless lift plugin</a></p><![CDATA[Understanding Codat Webhooks]]>/2022/04/codat-webhooks//2022/04/codat-webhooks/Fri, 01 Apr 2022 00:00:00 GMT<h2>What is a webhook?</h2> <p>As developers, we often find ourselves wanting to see exactly when an operation in an external API finishes or when a particular event happens within an external service. It would make sense to poll that API regularly to see if something has changed, checking in on all our operations until they’re complete. This polling can be taxing and really annoying to manage–which is the problem webhooks solve.</p> <p>Let’s look at one simple example of using a webhook: Github and Slack. When we create a new pull request in Github, if you’ve integrated Slack, you’ll automatically get sent a Slack message if you’re assigned as a reviewer on that pull request. This is a perfect example of utilizing a webhook–instead of constantly polling Github to look for new pull requests, Slack simply listens for the Github webhook to call its API, then parses that request to send messages to the reviewers–no polling required.</p> <p>Webhooks let us specify endpoints for our application that an external service will call when a particular event occurs. Whenever we are using a webhook, we need to specify to the external provider what type of event to notify us of and what API endpoint in our application we want this webhook to call. Once configured , we’ll get a request from the external service whenever that event occurs.</p> <h2>Why does Codat use webhooks?</h2> <p>Codat interfaces with a number of different accounting, banking, and eCommerce platforms. Each of these platforms have different rate limiting restrictions, and all these services have the potential to go down at any time. To deal with this, Codat’s API works asynchronously, so requests to Codat are acknowledged and given an operation key that can be used to get the status of that API call. So, when we start an operation in Codat, we don’t know immediately when that operation will complete. It might be a few seconds until the operation actually finishes–this is where we benefit from having a webhook to notify us that it’s finished.</p> <p>One of Codat’s most important webhooks is the “Push Operation Status Changed” event. When we get a call of this type, it tells us that the operation we started to save data to Codat has completed. (Or, if it encountered a problem, it will give us a detailed description of what went wrong.) The webhook is what allows Codat to asynchronously carry out our requests without making us poll over and over to see the status of those requests.</p> <p>What does this look like in practice? Let’s say our application has a table called “Expenses” and we want to sync this data as a directCost object in Codat. Additionally, we probably want to know when an expense has been successfully pushed. Using a webhook, our flow would be very simple: Create the directCost object based on a row in our Expenses table. Mark that expense as having a status of “Push in progress” and save the push operation key. When the push operation is complete, we’ll get a Codat webhook callback, and we’ll find our expense based on the push operation key we saved and update based on the push operation status: If it succeeded, we’ll mark our expense with “Push Complete”. If it failed, we’ll mark it with “Push Failed”. And that’s it–since we’re using a webhook, we never need to do any polling to get the status of our expense as it’s syncing.</p> <h2>Securing Codat Webhooks</h2> <p>If you’re using authentication for your backend, you’ll probably have to set up an exception for the Codat webhook endpoint. Codat offers two forms of authentication–basic auth with a set username and password and bearer authentication with a static token. Either way, you will have to check against these static credentials in your backend to make sure the webhook endpoint is available to Codat but not to the outside world.</p> <h2>Multiple push operations</h2> <p>Codat’s data types rely a lot on each other, so it’s not uncommon to be in a situation where you need to manage multiple sequential push operations.</p> <p>For example, let’s say you again wanted to push a directCost, but as part of that directCost, you wanted a reference to a new Account object that did not exist yet in Codat. To do so, you would have to go through an exchange like this:</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/c5c4df35502ecb418909b60c11368bc3/50383/codat-webhook-flow-example.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 107.56756756756755%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAWCAYAAADAQbwGAAAACXBIWXMAAAsSAAALEgHS3X78AAACCklEQVQ4y5WUh6oCQQxF9///y67YCxZs2Hvv5nECWUbdFd/AsJmSm5ubzHqdTkdWq5UsFgt/sh6PxzKZTITxfD51Ms7ns/R6PVmv1/79+XzuY3in00mNVCol9XpdCoWCgjG4FAS43W7V5lupVKRarYoND4YADgYDGY1GMhwOZTabqW0MH4/HB0OC2T0IGFPvcrnI9XpVJyLiwJoZNu73u5gfqZvN13Mv7vd7n8nxeNQ1QQ6Hg9rsbTYbQSYX3Hw0ZVtw0O12dQLSbrclk8lIPp+XbDarGpdKJYlGo5LL5RS42Wyq5pYNWD4gm2iBDlBfLpfKCkeXISkS0O40Gg3dCwScTqd+OhQLVlQxmUxKJBJRNul0Whnebje9h4/bDT4gX5xxgA3VI8VEIvECjAytVkuZ0wUExv4ARMN+v6/tQ8WZ3wYMkYD7oSkTEb0Y9BVrUmJi03dMtLZCsA5MmYikSdOa2KTM6ykWi/oaWPNFFoJwFo/HfRIfgNbx2C4jKg8AqRGUNQ8ArUM1tCpziYE21oexWEwn/UiRYIzmP1UZh91up2xJu1wu6z6Vxa7Valo82H6tMmnCiohUmLX7pKzq7MGOc9LmZ/K1yvZrsuH+ut4HQcgkMGUOYWgaGtC3aVq/pGyG/W1M7DBWLvMgH8+Nxjv+BfDdx9X3BZCn9F9AmvoF0HXm8L8pv/v8AYRbmZ0OxsdqAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Webhook Diagram" title="Webhook Diagram" src="/static/c5c4df35502ecb418909b60c11368bc3/50383/codat-webhook-flow-example.png" srcset="/static/c5c4df35502ecb418909b60c11368bc3/1d79a/codat-webhook-flow-example.png 185w, /static/c5c4df35502ecb418909b60c11368bc3/1efb2/codat-webhook-flow-example.png 370w, /static/c5c4df35502ecb418909b60c11368bc3/50383/codat-webhook-flow-example.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>Managing the dependencies between different objects that you want to push to Codat will require special backend logic.</p> <h2>Managing connection status</h2> <p>The first step after creating a company in Codat is commonly to add a connection to an accounting platform–and webhooks can notify us when that connection is established.</p> <p>The “Company data connection status changed” status is what notifies us when we successfully connect to an accounting platform like Quickbooks or Xero, and gives us the ID for that new connection. Many of the calls we will want to make to Codat will take in this connection ID in addition to the company ID, so storing this in the backend is a great idea–and getting it in a webhook makes that easy.</p> <p>This can also be helpful when we disconnect an application.If the user disconnects Codat from their accounting platform, getting notified by the webhook will be crucial, otherwise our application won’t know that the connection has been invalidated and we’ll get errors when we try to push data.</p> <p>It’s important to note that the Codat connection status changed webhook will not trigger automatically when Codat is disconnected on the accounting platform side. Only when Codat next attempts a sync will it realize the connection is severed and update the connection status for that platform.</p> <h2>Summary</h2> <p>It is crucial to understand webhooks when building applications with Codat. If you’re planning on using Codat to integrate with accounting, eCommerce, or banking platforms, you should be sure to include webhooks in your architecture’s design.</p><![CDATA[How to beautify java code reliably]]>/2022/03/beautify-java//2022/03/beautify-java/Wed, 09 Mar 2022 00:00:00 GMT<p>I recently had to set up code formatting on a spring boot java application. Auto code formatting is important to avoiding useless diffs in source files, reducing noise in code review, allowing reviewers to focus on what matters.</p> <p>Ideally we want:</p> <ul> <li>Automatic formatting on file save in IDE's (vscode, IntelliJ)</li> <li>Capacity to run in CI</li> </ul> <p>I wanted to integrate well with vscode and after trying some solution like:</p> <ul> <li>java autoformat, which I couldn't run in CI and that was not really portable between IDEs</li> <li>maven auto formatting plugin (didn't run on save easily in IDEs)</li> </ul> <p>I ended up, from a friend suggestion, looking into a <a href="https://github.com/jhipster/prettier-java">java prettier plugin</a>. Prettier allowed for an easy integration with most Web IDE, can be configured to format on save for vscode and is trivial to run in CI.</p> <h2>Installing prettier for java</h2> <p>Let's start with the setup in command line:</p> <ul> <li>Install nodejs: (<a href="https://github.com/Schniz/fnm">I highly recommend using a node version manager like fnm</a>) and to install a recent node version (<a href="https://nodejs.org/en/about/releases/">current long term support is 16+</a>)</li> <li>Add a <code>package.json</code> file at the root of your project and add prettier and the java plugin:</li> </ul> <div class="gatsby-highlight" data-language="json"><pre class="language-json"><code class="language-json"><span class="token punctuation">{</span> <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"api"</span><span class="token punctuation">,</span> <span class="token property">"version"</span><span class="token operator">:</span> <span class="token string">"0.0.1"</span><span class="token punctuation">,</span> <span class="token property">"private"</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span> <span class="token property">"devDependencies"</span><span class="token operator">:</span> <span class="token punctuation">{</span> <span class="token property">"prettier"</span><span class="token operator">:</span> <span class="token string">"2.5.1"</span><span class="token punctuation">,</span> <span class="token property">"prettier-plugin-java"</span><span class="token operator">:</span> <span class="token string">"1.6.1"</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token property">"engines"</span><span class="token operator">:</span> <span class="token punctuation">{</span> <span class="token property">"node"</span><span class="token operator">:</span> <span class="token string">"16.x"</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token property">"scripts"</span><span class="token operator">:</span> <span class="token punctuation">{</span> <span class="token property">"format"</span><span class="token operator">:</span> <span class="token string">"prettier --write \"{,**/}*.{java,yml,yaml,md,json}\""</span><span class="token punctuation">,</span> <span class="token property">"check-format"</span><span class="token operator">:</span> <span class="token string">"prettier --check \"{,**/}*.{java,yml,yaml,md,json}\""</span><span class="token punctuation">,</span> <span class="token property">"lint"</span><span class="token operator">:</span> <span class="token string">"npm run check-format"</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span></code></pre></div> <ul> <li>Install the required node_module: <code>npm install</code></li> </ul> <p>💡 Don't forget to add <code>node_modules</code> to your <code>.gitignore</code>.</p> <p>💡 <a href="https://prettier.io/docs/en/ignore.html">I highly recommend adding a <code>.prettierignore</code> that list autogenerated files</a> to avoid formatting those, eg:</p> <div class="gatsby-highlight" data-language="txt"><pre class="language-txt"><code class="language-txt">node_modules target</code></pre></div> <p>You can now format java code running <code>npm run format</code> 🚀</p> <p>💡 Do not forget to add node as a dependency for your project installation and to run <code>npm ci</code> in your project installation script for new developers.</p> <p>We now want to improve developer experience by integrating with IDEs, running the formatter in CLI being impractical.</p> <h3>Monorepo</h3> <p>If you are using a monorepo, I highly recommend setting up prettier in the root of your monorepo instead of setting up prettier in each sub directory.</p> <h2>Integrating with IDEs</h2> <h3>vscode</h3> <p>To set up prettier in vscode: add the prettier for java extension to recommended vscode extensions for the project, to do so, you can simply add the id of the extension to: <code>.vscode/extensions.json</code>:</p> <div class="gatsby-highlight" data-language="diff"><pre class="language-diff"><code class="language-diff">{ <span class="token unchanged"><span class="token prefix unchanged"> </span> "recommendations": [ </span> "pivotal.vscode-boot-dev-pack", "vscjava.vscode-java-pack", "redhat.vscode-xml", "sonarsource.sonarlint-vscode", "gabrielbb.vscode-lombok", "dbaeumer.vscode-eslint", <span class="token inserted-sign inserted"><span class="token prefix inserted">+</span> "esbenp.prettier-vscode", // the classic prettier extension </span> "shengchen.vscode-checkstyle", <span class="token unchanged"><span class="token prefix unchanged"> </span> ] </span>}</code></pre></div> <p>then set up this extension as the default linter for java file in <code>.vscode/settings.json</code> :</p> <div class="gatsby-highlight" data-language="diff"><pre class="language-diff"><code class="language-diff">{ // rest of your configuration // automatic format on save, I recommend the save on focus lost option as well <span class="token inserted-sign inserted"><span class="token prefix inserted">+</span> "editor.formatOnSave": true, <span class="token prefix inserted">+</span> "[java]": { <span class="token prefix inserted">+</span> "editor.defaultFormatter": "esbenp.prettier-vscode" <span class="token prefix inserted">+</span> } </span>}</code></pre></div> <p>Already we can easily format java code on save: <img src="/2a0959167a92c451097eb1d9f81b4eed/format-on-save.gif" alt="Formatting java code on save"></p> <p>I <strong>highly</strong> recommend checking in the <code>.vscode</code> folder in your versioning tool (most likely git) to share this configuration with all developers.</p> <h3>IntelliJ</h3> <p>To integrate with prettier java formatter:</p> <ul> <li>First install the <a href="https://www.jetbrains.com/help/idea/using-file-watchers.html#ws_file_watchers_bedore_you_start">File Watchers IntelliJ plugin</a></li> <li>Then configure the plugin <code>Tools/File Watchers</code>:</li> </ul> <div class="gatsby-highlight" data-language="txt"><pre class="language-txt"><code class="language-txt">Name: Prettier-java File type: Java Program: full path to `.bin/prettier` eg: `node_modules/.bin/prettier` Arguments: --write $FilePathRelativeToProjectRoot$ Output paths to refresh: $FilePathRelativeToProjectRoot Auto-save edited files to trigger the watcher: check it Trigger the watcher on external changes: check it</code></pre></div> <p><a href="https://prettier.io/docs/en/webstorm.html#running-prettier-on-save-using-file-watcher">prettier documentation can be found here for IntelliJ integration</a></p> <h2>Integrate code formatting check in CI</h2> <p>Now we only need to integrate this check in CI, I'm using gitlab for this example, but the principles can be transposed to any other CI.</p> <p>First we need to add a new job:</p> <div class="gatsby-highlight" data-language="yaml"><pre class="language-yaml"><code class="language-yaml"><span class="token key atrule">stages</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> build <span class="token punctuation">-</span> test <span class="token punctuation">-</span> deploy <span class="token comment"># ... rest of the file</span> <span class="token comment"># beautify backend files with prettier</span> <span class="token key atrule">lint-back</span><span class="token punctuation">:</span> <span class="token key atrule">image</span><span class="token punctuation">:</span> node<span class="token punctuation">:</span>lts<span class="token punctuation">-</span>alpine <span class="token comment"># We use a node image to run prettier</span> <span class="token key atrule">stage</span><span class="token punctuation">:</span> test <span class="token comment"># must match an existing stage</span> <span class="token key atrule">needs</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token punctuation">]</span> <span class="token comment"># none required optimization that explain to gitlab ci that this test has no dependency and can start as soon as possible</span> <span class="token key atrule">script</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> npm ci <span class="token punctuation">-</span> npm run lint <span class="token comment"># ... rest of the file</span></code></pre></div> <p>now just push your change, and you are all set:</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 569px; " > <a class="gatsby-resp-image-link" href="/static/af9cf2ab16141324bb84ae6caad39b06/854dc/CI_success.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 63.78378378378379%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAIAAAAmMtkJAAAACXBIWXMAAAsSAAALEgHS3X78AAABmklEQVQoz41Sa5OjIBD0+8nDrJIgrqA8RNDE7P//cdfRSnY3VVd1XVNU8+hhpqEoQ0kcoWdKCQU452wH+IsQQtgT1Q8U8St2vtOj9t477y6XyziOwzBM0zTuwHpKqe/7zx3I/i2+L/fZz5Ob1rTeb/cQQs7ZOde2LRK1O6AEP+/4dbPIog51bWuRxNmelVSqUzj01gJ/4pdYztLebOvaxjVqVJzxsizfDv0LRQppzasdrLc++GCMQZF1Xf+PvpjitKyLtRbdzvMMn9Z1hWGo9lX2G155C2i2bYMYfUIMq8BBuq6DVbC323FYDXKY97r5AWiwl3LGVEophMAJpRSeShujNYYHQLCCrePyIse4pBRDSDFmlO39qDVeSUl5FqL5+LgIAYKxQsX4LbCTkOoQq5jMcvXrTY3+c54RGl9iWcZtMzH2Kdlt0zkP12s9DFxKrjU3pmoa6AuyuDKNf5Ihc0/tQLWhStFhIMbQriMgbUv7HoQ5x/ueWctCQJaHmPuSO8KWkt1IVdMTZSfOT4w94iDPaUVpdYyIvey/b69NeHMZbQMAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="CI running lint successfully" title="CI running lint successfully" src="/static/af9cf2ab16141324bb84ae6caad39b06/854dc/CI_success.png" srcset="/static/af9cf2ab16141324bb84ae6caad39b06/1d79a/CI_success.png 185w, /static/af9cf2ab16141324bb84ae6caad39b06/1efb2/CI_success.png 370w, /static/af9cf2ab16141324bb84ae6caad39b06/854dc/CI_success.png 569w" sizes="(max-width: 569px) 100vw, 569px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p><![CDATA[Choosing the best storage solution in GCP]]>/2022/03/choosing-gcp-storage-solution//2022/03/choosing-gcp-storage-solution/Wed, 09 Mar 2022 00:00:00 GMT<p>Are you new to GCP, looking to make sense of all the storage offerings GCP provides or simply hoping to learn something new? You're in the right place!</p> <p>In this blog we're going to compare the key differences between GCP's numerous storage solutions. We will start by taking a look at a handy decision tree which gives you an overview of the various options. We will then categorise the available storage solutions into blob, relational, NoSQL, file, block and other storage. For each option, we will delve a bit deeper into the specifics and we will highlight the best suited use cases as well as flag any potential downsides. Let's get started!</p> <hr> <h2>Decision tree</h2> <p>As promised, we're starting with a decision tree to help you choose the best storage tool for your use case. Don't worry if you're not sure what all this means - we'll take a look at each tool below.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/d0556336001f0ccebbe3429adfaa6b90/50383/decision_tree.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 224.86486486486487%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAtCAIAAAAhntw/AAAACXBIWXMAAAsTAAALEwEAmpwYAAAGfUlEQVRIx5VWXYgkVxW+vhoQX0TxXd+iAUUCvgsawhpDiIIoCBqMrJjgg0Z8UJCYzUoMrGDUSdYlMT5kE3c3kdXsMOMsuzO7s5me393umenp+enuureq7v9/VXW3p3p2EhZmJgrfQ9fturfO+c53vnORi0Usq1BW0gZu/P8F5GOprfMhKheodsz4fRyzh5kCwE1EJpQOEEvhgrYxuiK6EqIQR8F4oY0whmuLci73uv1eQriyqyQ7MfvujxqzfSYJUwlT+B6IhIcMr9hr94vJT7q1J5A0jjsnjFXW96V+sbXy6tYGh/gPga2jZXu2+TOz+qTenkAylqyXSmW5i9L4YRxWoTqWp8j8iIURc0OUL9zpPXwyefYv3NVsE2VTbY/d7LgWXHOuFeLSJjdW860efBnIKHwZXME+9MsAN0CQXk8pLBUwnGv3TndnJulLEw7/JuQsidl5xXb+oJN/IajxRmsDY6KMX8TkwelL37ox1TuS7ZgljfCfT+nLH4nL30SmqNKqEoMB1BAIy5QFYoU5Sm2Bay3YlqAbXPQQ20nwmb9nl69xGyBVEBmtw9tX0gfbYB2S+uCI8ekoX9/p/fhZfPYC91FoX/oSRMZ1/ZJyoNm4fwrIrvKV2CcsHJTKlBUdVGowAMIybS/ubU/XhHmm7G4Pd/tE1Fn4qzi5sNtJdeCiJsxsAWGXUYhFhlMlFRC2kOAvTl14bG6yy2TO1fpmu7W+yaTuUfntm9Ofnzw/T7jADT95n76EYuMEkqGg1UBUNWHGxhykpr2yQYeSVlVeliaWatwPEJcZZyG0lEYCc4jt4t7pv6bvXBU+0nGEUE9mPRcGn71IJt6iTAFVtD7UZrpgrONWfmAbj9mtF1De2u49/Tw5dxEUBkWiYzBgnsrktxP4N39OE5pw3YdHKvrcZ+SOvvkVPvMFaA8kizKNBT8IO7gy+nGcoVCjkQSEQoPdBFgvnC+FFiabdem0pGvIOC+lElLV/ZxCP1852bgOCqNSb23vbm/vcWn6TD29OPfQtcuLmeK44a7cpy6hsHACNgM3UftaXn2hf9dcPtdeh/gltHdCMM7gB+j/b1sbp243ujIK2gxzD8SZT7u1J5HwRbJHKDd1P9uDfgbmfJGXwHYFYYNaYXFUDHXNdhSuUr5ipkT5Uqv/6FPk9Fk+LkM+JgykCv2anD6Hn3uZZiJXLhUG5JKpguctefNrePpB33oGwUt4bjnf3NvX9vubGVNk4k3ypzfSlBHYCY9cYRHybIO9993e1a/bzeeQdCELUQBs7Z6jMBiE2sOh7PKAbQkZHUBYyM4ZX7sNAh9ttjb7mGgbWhkF9zzZmE2lyYRJmcoAwhxApbJkZKmY+UT4N4rLjyNtXKezk6Y5sN2h/NSdpYl2E9JLD4EiEsJu+fe+aq5/ybV+gVQss6IU4Ae16YfKl2Bjx5m+jdyNAMwOEOumycv/SKfm9wnL7mn6o9xTj2EQbW73vv8rYJX74q62oTGMu3c4+buav3cdiVCSEGlRwZeti0A16AG0ze9ibCm2FskwVPACt1H4kQgQ+QAJZWBWpRlo2d3J6OM3pn66dCPhikmTkIzkFJwEc/3z5flH595dy5VI1+z1B/QUyPOHSFnXJzmMO2B7m4lfrtw6s76aSZiDjqSUwKHWw+MfN24/s3SzDSqmbb34HTX/sNk8BWEXIB1WcxjAKEZxOIgDOAjWs2oAAJGADQ6K4agcWTBJV4pyJKoRiyOUL6/3H/kJBm372gzA+wF1tkwmZ17HL76WE0Y42JjYyzmM2Jws25nPmisf00vfQyyXePpWfrvDx0blXGEhCiib0OTM6+T3r1Iqa6qloUpTXbKsaee+7K5+Rq89hZSPfWM5aNsEGBdzhCzlOVNQME+HQzYa1Zb0/nwG65WZ7p5XO68pPIOsD+12h1L4Kyxh8rnJtx6ZvQI3AxA2zgUBQD/dhcA85smCn/q4/icKi99A+mBWQXuBKneFhPuBOlqbQiuZr6hsQbA2Yv0M5JlN39p3T2XAj8Kx8zkwO2Qgb1shutbuPfFr/NIbIB3+P2pbCa44VwrJsszG1svHlzhoKf9hNwPQpoggz+FYnj2cU67Gd5JbabqWg6oOjRzsqRAS882XWPMF1XsbMaFWm+u7/URp30jI/WO2u/SwmwGFm0GR9+eTNz+6/grScw/V10cMNqItFByO73DZFUocla2JXOZs9+1s67zAs/8FkBuI5ptn56gAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Decision tree" title="Decision tree" src="/static/d0556336001f0ccebbe3429adfaa6b90/50383/decision_tree.png" srcset="/static/d0556336001f0ccebbe3429adfaa6b90/1d79a/decision_tree.png 185w, /static/d0556336001f0ccebbe3429adfaa6b90/1efb2/decision_tree.png 370w, /static/d0556336001f0ccebbe3429adfaa6b90/50383/decision_tree.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>Now let's look at each of the options listed above in a bit more detail.</p> <hr> <h2>Blob storage</h2> <p>Blob storage allows you to store unstructured object data. This means that we're looking at data which often consists of items (e.g. media files) which don't need to sit within a file structure.</p> <h3>Cloud Storage</h3> <p>We're starting with one of the most commonly used storage options - Cloud Storage. Cloud Storage is GCP's blob storage solution and it is often used alongside other tools such as Compute Engine (service for creating and running virtual machines), App Engine (platform for building highly scalable applications using serverless) and BigQuery (data warehousing solution - more info below). In terms of its structure, it uses buckets to group items.</p> <p>Benefits</p> <ul> <li>Buckets are really powerful organisation tools, which allow you to control lots of properties, including lifecycle policies (e.g. delete object 2 years after creation) and object versioning (by default, object versioning is switched off, so if updated, the object will be overwritten).</li> <li>You can choose to either host your bucket in a single region, or it can be dual-regional or multi-regional. The dual-regional option in particular isn't seen across many storage options. It can be a great way of saving on costs (in comparison to multi-regional buckets) whilst bringing your data closer to your users and making sure it is available even in the event of a regional outage.</li> <li>You can choose a suitable storage class based on how often you need to access your data. The four storage classes available are standard (access regularly), nearline (access once a month), coldline (access once a year) and archive (access less often than once a year), with the cost of storage decreasing with each subsequent class.</li> </ul> <p>Downsides</p> <ul> <li>Cloud Storage cannot mimic file structure of nested folders. Each item is accessed directly via its own url.</li> <li>Coldline and archive classes carry an associated retrieval cost, which will be charged when you access your data. This means that the costs can sneak up on you, especially if you’re not keeping track of how often you’re accessing objects in your long-term storage buckets.</li> </ul> <p>In terms of common use cases, Cloud Storage is an excellent option for archiving data for audits and for storing data for disaster recovery. It is also great for minimising latency for users who download large data objects since you can choose dual-regional or multi-regional storage to bring your data closer to your users.</p> <hr> <h2>Relational storage</h2> <p>Unlike with blob storage, relational databases organise data into tables which can be linked to each other. They are the industry standard and GCP provides a couple of options for storing relational data within the cloud.</p> <h3>Cloud SQL</h3> <p>Cloud SQL is the obvious choice when setting up a relational database in GCP. It provides a fully managed service which now allows you to create instances of MySQL, PostgreSQL as well as Microsoft SQL Server.</p> <p>Benefits</p> <ul> <li>Easy to migrate to from on-premise relational databases, either using MySQL dump or CSV file imports.</li> <li>Variety of methods for connecting to your database, including access via private IP and Cloud SQL proxy.</li> <li>Automatic data replication with failover.</li> <li>Automated storage capacity management.</li> </ul> <p>Downsides</p> <ul> <li>Storage capacity is limited to 30 TB, which may catch some users out.</li> <li>Cloud SQL is great for vertical scaling, but if you're interested in horizontal scalability, you'd best look elsewhere.</li> </ul> <p>Cloud SQL is an excellent option if you're looking to replicate your on-premise database within the cloud, or if you'd simply like to stick with a database you know already. This is exactly what happened with AutoTrader, a digital automotive marketplace. They were looking to move their on-premise infrastructure to the cloud, and they chose to move their existing Oracle database into Cloud SQL.</p> <h3>Cloud Spanner</h3> <p>GCP’s alternative to Cloud SQL is Cloud Spanner. Cloud Spanner provides a relational database structure with the added advantage of horizontal scaling.</p> <p>Benefits</p> <ul> <li>If you're concerned that Cloud SQL doesn't provide enough capacity for your needs, Cloud Spanner is definitely worth a consideration since it scales to PB capacity.</li> <li>Support for Google Standard SQL as well as PostgreSQL.</li> <li>It provides strong consistency, meaning that once a user makes a change to the data, the update will be consistent for all users immediately.</li> <li>It automatically replicates data across multiple zones.</li> </ul> <p>Downsides</p> <ul> <li>Whilst you can create backups on demand, there is no in-built option to generate them automatically.</li> <li>Cloud Spanner isn't MySQL compliant.</li> </ul> <p>Since Cloud Spanner offers strong consistency globally, it is well suited to financial applications. A fun real-life example is Pokemon Go, which uses Cloud Spanner due to the aforementioned feature.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 640px; " > <a class="gatsby-resp-image-link" href="/static/b5397843fe90a2cd8e538a0e19d8bb6f/c08c5/pokemon_go_unsplash.jpg" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 80%; position: relative; bottom: 0; left: 0; background-image: url('data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAQABQDASIAAhEBAxEB/8QAGAAAAwEBAAAAAAAAAAAAAAAAAAECAwT/xAAUAQEAAAAAAAAAAAAAAAAAAAAB/9oADAMBAAIQAxAAAAFRrxDAwf/EABoQAAEFAQAAAAAAAAAAAAAAAAEAAgMRIiH/2gAIAQEAAQUCqwaTumLcTX4X/8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAwEBPwE//8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAgEBPwE//8QAGhAAAgIDAAAAAAAAAAAAAAAAADEBEQIQEv/aAAgBAQAGPwK1G8+WVSEf/8QAGxABAAIDAQEAAAAAAAAAAAAAAQARITFBcVH/2gAIAQEAAT8hMl4RGkhbdewdkYB3L7h9VuZ6p//aAAwDAQACAAMAAAAQUy//xAAYEQEBAAMAAAAAAAAAAAAAAAABABEhMf/aAAgBAwEBPxBcO+QF/8QAGBEAAwEBAAAAAAAAAAAAAAAAAAEhETH/2gAIAQIBAT8QS1TpT//EABwQAQEAAgIDAAAAAAAAAAAAAAERACExQZGhwf/aAAgBAQABPxALCDFyqYHDy9fMTBB0DeUo1Upw3PWacFxeC4heAxGWkz//2Q=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Pokemon Go" title="Pokemon Go" src="/static/b5397843fe90a2cd8e538a0e19d8bb6f/c08c5/pokemon_go_unsplash.jpg" srcset="/static/b5397843fe90a2cd8e538a0e19d8bb6f/d7fe6/pokemon_go_unsplash.jpg 185w, /static/b5397843fe90a2cd8e538a0e19d8bb6f/f4308/pokemon_go_unsplash.jpg 370w, /static/b5397843fe90a2cd8e538a0e19d8bb6f/c08c5/pokemon_go_unsplash.jpg 640w" sizes="(max-width: 640px) 100vw, 640px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <hr> <h2>NoSQL storage</h2> <p>By comparison to relational storage, NoSQL databases store data in non-tabular formats. This can include document storage, key-value pairs or wide-column databases. GCP offers two main NoSQL databases, which we will explore below.</p> <h3>Cloud Bigtable</h3> <p>Cloud Bigtable consists of sparsely populated columns, where rows are indexed by a single key. It is possible to group columns into column families to capture which columns are related. Each row-column intersection can contain multiple cells at different timestamps, and this allows Cloud Bigtable to record a history of data updates.</p> <p>Benefits</p> <ul> <li>It can comfortably handle PB capacity.</li> <li>It offers very low latency, consistently hitting targets of sub-10ms.</li> <li>Easy integration with big data tools, including Hadoop and BigQuery.</li> <li>Clusters can be resized with no downtime.</li> </ul> <p>Downsides</p> <ul> <li>The smallest possible cluster consists of 3 nodes and provides 30,000 operations per second.</li> <li>Cloud Bigtable doesn't scale to zero, so you pay for nodes regardless of whether they are being used.</li> </ul> <p>Cloud Bigtable works well for operational and analytical use cases, and is often used with IoT applications. A real-life example is Dow Jones, a news content and business information provider, who uses Cloud Bigtable to store their data before processing it for analytics.</p> <h3>Firestore (previously Datastore)</h3> <p>Previously known as Datastore, Firestore is the new generation of this NoSQL document database. Unlike Cloud Bigtable, it uses JSON to store data.</p> <p>Benefits</p> <ul> <li>Integration with Firebase - GCP's app development offering.</li> <li>Firestore offers strong consistency. This is a typical feature of relational databases.</li> <li>Multi-regional replication is available.</li> <li>It provides ACID transactions, allowing you to be confident that there are no transactional discrepancies within your data.</li> </ul> <p>Downsides</p> <ul> <li>It only scales up to TB capacity.</li> </ul> <p>Firestore is an excellent choice for mobile and web applications at global scale. Thanks to its integration with Firebase, setting up Firestore can be a seamless experience. An example of Firestore being used in the real life is The New York Times, who use Firestore to support their real-time rich text editing tool.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 640px; " > <a class="gatsby-resp-image-link" href="/static/4170bae72f333f348d45c0304674efe4/c08c5/new_york_times_unsplash.jpg" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 66.48648648648648%; position: relative; bottom: 0; left: 0; background-image: url('data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAANABQDASIAAhEBAxEB/8QAGAAAAwEBAAAAAAAAAAAAAAAAAAIEAwX/xAAVAQEBAAAAAAAAAAAAAAAAAAACAf/aAAwDAQACEAMQAAABdufUVgTEX//EABoQAAIDAQEAAAAAAAAAAAAAAAIDAAEEERP/2gAIAQEAAQUCyBXi8FiEDSYLYfU3c//EABURAQEAAAAAAAAAAAAAAAAAAAAR/9oACAEDAQE/AVf/xAAUEQEAAAAAAAAAAAAAAAAAAAAQ/9oACAECAQE/AT//xAAbEAEAAgIDAAAAAAAAAAAAAAABAhEAEBIxMv/aAAgBAQAGPwJZRG3FI64lVnXrX//EABsQAQADAQADAAAAAAAAAAAAAAEAESExQWFx/9oACAEBAAE/Id9C6eCU8G6hYZnqGiqMzZ8wOvIgz//aAAwDAQACAAMAAAAQGB//xAAYEQACAwAAAAAAAAAAAAAAAAAAEQExUf/aAAgBAwEBPxCLbH0//8QAFxEBAQEBAAAAAAAAAAAAAAAAAQARUf/aAAgBAgEBPxBeFl//xAAeEAACAgICAwAAAAAAAAAAAAABEQAxIUFhgVFxof/aAAgBAQABPxAHEEKOBYfMMcVAMhNjer+GIgSXQp6hapQsA7hOYLPLv3BAClP/2Q=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="New York Times" title="New York Times" src="/static/4170bae72f333f348d45c0304674efe4/c08c5/new_york_times_unsplash.jpg" srcset="/static/4170bae72f333f348d45c0304674efe4/d7fe6/new_york_times_unsplash.jpg 185w, /static/4170bae72f333f348d45c0304674efe4/f4308/new_york_times_unsplash.jpg 370w, /static/4170bae72f333f348d45c0304674efe4/c08c5/new_york_times_unsplash.jpg 640w" sizes="(max-width: 640px) 100vw, 640px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <hr> <h2>File and block storage</h2> <p>We are now moving on from nonSQL storage to file storage, by which we mean storage which allows us to replicate the commonly used file structure of nested folders. The best way of thinking about it is that it mimics the file system of a computer. Block storage, by comparison, stores data as separate pieces, where each piece can be uniquely identified. Block storage is very performant since it allows for data to be accessed via multiple paths.</p> <h3>Filestore</h3> <p>Filestore (not to be confused with Firestore) is GCP's fully managed network attached storage (NAS) option. It is often used alongside instances of Compute Engine and Kubernetes Engine. Filestore also offers four tiers (basic HDD, basic SSD, high scale SSD and enterprise) to allow you to pick the best solution for your needs.</p> <p>Benefits</p> <ul> <li>Filestore offers high performance - throughput of 25 GB/s and 920,000 operations per second.</li> <li>It scales up to 100 TB capacity.</li> <li>Capacity automatically scales up and down to minimise cost.</li> </ul> <p>Downsides</p> <ul> <li>It is rarely used without an associated Compute Engine or Kubernetes Engine instance.</li> </ul> <p>Filestore is often used for file sharing as well as high performance computing, e.g. genomics processing.</p> <h3>Persistent disk</h3> <p>We are now delving into the world of storage solutions for Compute Engine. GCP provides a variety of options within the persistent disk offering. As expected, at the the highest level of categorisation, you can choose between SSD and HDD.</p> <p>Benefits</p> <ul> <li>Automatic encryption using GCP-supplied keys or your own.</li> <li>Durable storage - there is very low risk of loss of data, with the risk lower for regional persistent disks.</li> <li>There is no charge for operations.</li> <li>The disk types include standard, balanced, extreme or SSD, so you have a lot of freedom to choose the best suited alternative.</li> </ul> <p>Downsides</p> <ul> <li>Its usage is primarily restricted to Compute Engine.</li> <li>A persistent disk can also be used with Kubernetes Engine as storage for persistent volumes.</li> <li>Persistent disks can be zonal or regional, so they aren't well suited for global applications.</li> </ul> <p>The most common use case for persistent disks is, unsurprisingly, attaching them to Compute Engine instances.</p> <h3>Local ephemeral disk</h3> <p>The final file storage type we'll cover is local ephemeral disk. The main difference between local ephemeral disks and persistent disks hides in the name - persistent disks retain data even if they are not attached to a virtual machine and the data will be kept until the disk is deleted. By comparison, the data stored on local disks is lost as soon as the virtual machine it is attached to is stopped or deleted.</p> <p>Benefits</p> <ul> <li>Local disks provide higher throughput and lower latency than persistent disks since they are physically attached to the server.</li> <li>It is possible to attach up to 9 TB per instance using a maximum of 24 local disks.</li> </ul> <p>Downsides</p> <ul> <li>Local disks are only available as SSD, so HDD local disks don't exist.</li> <li>You can't attach local disks to certain virtual machine types.</li> <li>Whilst you can be confident that your disk is always encrypted using GCP-supplied keys, you can't supply your own encryption keys.</li> </ul> <p>Local disks are best suited if you need fast scratch disk or cache.</p> <hr> <h2>Other storage types</h2> <p>In this final section we introduce the remaining two storage solutions which don't fit well into any of the categories mentioned above.</p> <h3>Memorystore</h3> <p>Memorystore is GCP's fully managed service for Redis and Memcached. If you are interested in using either Redis or Memcached but don't want the hassle of managing them, Memorystore is the storage option for you.</p> <p>Benefits</p> <ul> <li>Memorystore provides sub millisecond latency for lightning fast data access.</li> <li>It is easy to migrate from an existing Redis or Memcached instance using the lift-and-shift functionality.</li> <li>Memorystore natively integrates with GCP's IAM service for robust security.</li> </ul> <p>Downsides</p> <ul> <li>Memorystore's capacity is limited to 300 GB.</li> <li>If cost is a significant factor, then you should consider hosting your own Redis or Memcached instance.</li> </ul> <h3>BigQuery</h3> <p>GCP's data warehouse solution, BigQuery can handle data storage as well as analysis. It isn't the most common storage solution, and is often used alongside other storage options for data analysis only.</p> <p>Benefits</p> <ul> <li>BigQuery is great for data analysis - all your data can be in one place so you don't have to worry about data ingestion.</li> <li>It provides many specialised services, including BigQuery ML for machine learning and BigQuery BI Engine for interactive data analysis.</li> <li>You can also stream data directly into BigQuery from other storage solutions.</li> </ul> <p>Downsides</p> <ul> <li>Cost is relatively high, so it may not be the most efficient long-term solution.</li> <li>It's not recommended to use BigQuery for data storage unless you want to use its analysis capabilities as well.</li> </ul> <p>One of GCP’s customers using BigQuery is The Home Depot, who use this tool to keep track of their stock across 50,000+ items and 2,000+ locations.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 640px; " > <a class="gatsby-resp-image-link" href="/static/701fe909ec8d9de0b37f2a5910660941/c08c5/hardware_store_unsplash.jpg" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 66.48648648648648%; position: relative; bottom: 0; left: 0; background-image: url('data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAANABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAIBA//EABYBAQEBAAAAAAAAAAAAAAAAAAIAA//aAAwDAQACEAMQAAAB5Te5uAb/xAAZEAACAwEAAAAAAAAAAAAAAAAAAQIRIUH/2gAIAQEAAQUCoqJgtfD/xAAWEQEBAQAAAAAAAAAAAAAAAAAAARH/2gAIAQMBAT8BjH//xAAWEQEBAQAAAAAAAAAAAAAAAAAAARH/2gAIAQIBAT8BrX//xAAZEAACAwEAAAAAAAAAAAAAAAAAIQECEBH/2gAIAQEABj8Cng2KuPP/xAAdEAACAgMAAwAAAAAAAAAAAAABEQAhMVFhcYHR/9oACAEBAAE/IQC71WJQSB2LgRu9L7AYGbV6mTyVQHzfZ//aAAwDAQACAAMAAAAQgw//xAAXEQADAQAAAAAAAAAAAAAAAAAAAREx/9oACAEDAQE/EMjhn//EABcRAAMBAAAAAAAAAAAAAAAAAAABEVH/2gAIAQIBAT8QqjTT/8QAHBABAAMBAQADAAAAAAAAAAAAAQARMSFRQXGB/9oACAEBAAE/EO6IBsXD8BeezcadBnux1Kd9pv8AfpLrBVA6l807ITeXEqxGmlP/2Q=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Hardware store" title="Hardware store" src="/static/701fe909ec8d9de0b37f2a5910660941/c08c5/hardware_store_unsplash.jpg" srcset="/static/701fe909ec8d9de0b37f2a5910660941/d7fe6/hardware_store_unsplash.jpg 185w, /static/701fe909ec8d9de0b37f2a5910660941/f4308/hardware_store_unsplash.jpg 370w, /static/701fe909ec8d9de0b37f2a5910660941/c08c5/hardware_store_unsplash.jpg 640w" sizes="(max-width: 640px) 100vw, 640px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <hr> <h2>Conclusion</h2> <p>GCP provides a massive variety of data storage tools for all sorts of use cases. Selecting the best option can be difficult, since there are so many tools to choose from. My best advice is to start from your data type, this will often significantly narrow down the available possibilities. Afterwards, considering the location of your data (e.g. zonal or global), required consistency and capacity will help you decide. Make sure to consider cost as well - you can use GCP's pricing calculator to get a sense of how expensive different solutions would be.</p> <p>I hope that you're this leaving this blog post with a clearer idea of the different storage options GCP provides. Remember that if none of the tools listed here fits your use case, you can always build your own solution on Compute Engine. Best of luck with your cloud journey!</p> <h2>Useful links</h2> <p><a href="https://cloud.google.com/products/storage">Intro to storage options</a></p> <p><a href="https://cloud.google.com/blog/products/databases/how-auto-trader-migrated-its-on-prem-databases-to-cloud-sql">Cloud SQL for AutoTrader</a></p> <p><a href="https://cloud.google.com/blog/topics/developers-practitioners/how-pok%C3%A9mon-go-scales-millions-requests">Cloud Spanner for Pokemon Go</a></p> <p><a href="https://cloud.google.com/customers/dow-jones">Cloud Bigtable for Dow Jones</a></p> <p><a href="https://www.youtube.com/watch?v=LEoWPdQh27c">Firestore for The New York Times</a></p> <p><a href="https://cloud.google.com/customers/featured/the-home-depot">BigQuery for The Home Depot</a></p> <p><a href="https://cloud.google.com/products/calculator">Pricing calculator</a></p> <p><a href="https://unsplash.com/">Images from Unsplash</a></p><![CDATA[How AnotherDay and Theodo built a big data geospatial analytics platform]]>/2022/03/anotherday-big-data-geospatial-analytics-platform//2022/03/anotherday-big-data-geospatial-analytics-platform/Wed, 02 Mar 2022 00:00:00 GMT<p>AnotherDay’s Cascade platform provides businesses with the ultimate risk and intelligence management platform. With risk assessment at the heart of so many international standards, many business suffer from competing standards for tracking and mitigating risk. Often, large spreadsheets, unwieldy slide decks and or folders with hundreds of sensitive intelligence files become impossible to manage with challenges such as version control and centralisation. Cascade provides both a single centralised platform to collate intelligence and assets and also a powerful analytics platform to make sense of this data and to extract effective risk mitigation strategies.</p> <p>Today, an often used strategy for asset risk quantification is to run accumulations. This is where a radius is selected and for each of the businesses assets, the total value of all the other assets lying within that radius from the asset is calculated. A business can then look for where most value has accumulated, and therefore where they have incurred the most financial risk. This form of analysis is often used in areas such as terrorism modelling.</p> <p>The Cascade platform needed the ability to run these sorts of accumulations across large datasets, in excess of 1 million assets. As these modelling runs were potentially long-running it was also desirable that these tasks could be run asynchronously from a queue.</p> <p>AnotherDay worked with Theodo to build this functionality into the Cascade platform. As experts in React, Django and AWS the task was to make this end to end functionality production ready in 5 weeks.</p> <p>The modelling flow begins by uploading the assets into the Cascade platform. The data was provided in CSV format which was uploaded to S3 via the frontend. This triggered a lambda function which performed validation and cleaning of the data before leveraging the <code>aws_s3</code> extension for RDS PostgreSQL to transfer the data directly into RDS from S3.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/1408b34f355415d25ada3136adcecfe3/50383/asset-upload-flow.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 24.324324324324326%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAFCAYAAABFA8wzAAAACXBIWXMAABYlAAAWJQFJUiTwAAAA1ElEQVQY062Q0UoCURCGzzl7Drurg2tKrpsuuEoRq3SjiOJNgRlhT1A3vUVEL6Av0EXXvefXwUT0ui6+gfmZYf5/lFKKf8YX63Zojw0sYcWxeC4Z3uUYY4ii0A8GXLYcny8t1qX1vfV6hNaKWmPA7HFLVtx63S/owQhTTnCdHvWKUE8TXr+XPLyPqVaFi6yNjWssrwS+5myeMnQotNOUODJ0i3veNjBefPw6dN0+yc0UJcnB+vn1GdKJT+KIM6yGOb2mnOhBIBTFyh/P95GP0fpPP/wBtcF3A99cposAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Asset upload flow" title="Asset upload flow" src="/static/1408b34f355415d25ada3136adcecfe3/50383/asset-upload-flow.png" srcset="/static/1408b34f355415d25ada3136adcecfe3/1d79a/asset-upload-flow.png 185w, /static/1408b34f355415d25ada3136adcecfe3/1efb2/asset-upload-flow.png 370w, /static/1408b34f355415d25ada3136adcecfe3/50383/asset-upload-flow.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span> <em>Asset upload flow</em></p> <p>To store the geospatial data in RDS, Theodo used the PostGIS extension for PostgreSQL. A second RDS instance was provisioned alongside the existing database for Cascade in order to protect the main database from the expensive modelling queries.</p> <p>To trigger an accumulation modelling run, a user creates an accumulation entity in the frontend specifying parameters such as radius. Saving this entity triggers a JSON message sent to an SQS Queue with the model parameters and run details.</p> <p>Celery (an asynchronous task runner) was provisioned in Fargate, providing a worker pool that subscribes to the SQS queue. The model task is picked up by a celery worker which then runs the model SQL against the PostGIS database. A naive implementation of the model scales as N<sup>2</sup> with an analysis of 20,000 assets taking 15 minutes. Leveraging the r-Tree index provided by PostGIS provides blazing fast performance, lowering the model runtime to 15 seconds.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/c8a5969c2839da4c27329047214adefe/50383/running-an-accumulation-model.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 28.10810810810811%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAGCAYAAADDl76dAAAACXBIWXMAABYlAAAWJQFJUiTwAAABBUlEQVQY04WQyUrDUBSG75TbxJQOiaGxbQoptChBkCJ04bgQre4q6spFduKu2+76Cj6Ab+J7+DifCUHBSuviwPk5wz8IIQSbSv7BcuNu2J3Qjg/Lfm2oJdLoH+wpw6TZI3C8X3vaKKSSOMbgWJd88ck8/yhmeymqmxJGEbZmuXl+4PT+Ds918cMWF/EY8ndex+e4QYO61+DxJWP5dkWvHxOFu1hrGQ7nJMkM0T65Zic7RutK1WB/RH+UVhYdTcepszyYkfkdRKFKK00Q+SRpVNwoms0Wrudylg+ZPg0Kh2tZaalQW7KqSqKKKL6xKqzfrqZcLo5KLBH/PCgJ5FYCgakZtNV8AWPOjhEXO0/cAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Running an accumulation model" title="Running an accumulation model" src="/static/c8a5969c2839da4c27329047214adefe/50383/running-an-accumulation-model.png" srcset="/static/c8a5969c2839da4c27329047214adefe/1d79a/running-an-accumulation-model.png 185w, /static/c8a5969c2839da4c27329047214adefe/1efb2/running-an-accumulation-model.png 370w, /static/c8a5969c2839da4c27329047214adefe/50383/running-an-accumulation-model.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span> <em>Running an accumulation model</em></p> <p>The top 50 results (in terms of accumulated value) are saved to the main database for display in the frontend. Theodo used the Mapbox API to build an interactive results map, displaying the numbered results and the accumulation radius. For the rest of the results the <code>aws_s3</code> extension is used again to output a CSV to S3 which allows the entire result set to be downloaded.</p> <p>Focusing on system design tailored to the business domain, and leveraging AWS &#x26; open source tools, sets AnotherDay up to create a new standard for risk management. Theodo ensured best practices were followed to create a state-of-the-art platform with many exciting avenues to explore around integrating live intelligence data into the modelling process.</p><![CDATA[Architecting a Modern Monorepo with NX and Turborepo]]>/2022/02/architecting-a-modern-monorepo//2022/02/architecting-a-modern-monorepo/Tue, 01 Mar 2022 00:00:00 GMT<h1>What is a Monorepo?</h1> <p>A monorepo is a consolidated repository containing the source code of multiple projects, which are commonly managed by independent teams and also often share common packages.</p> <h2>Not just Code Collocation</h2> <p>Recently a common software development practice is to have a full stack JavaScript application, this leads to many developer teams believing they have a Monorepo "Because all of my code is in the same repository".</p> <p>A monorepo is far more than just code collocation, we'll address the benefits, drawbacks and showcase two industry leading Smart Build Systems.</p> <h2>Benefits of a Monorepo</h2> <p>There are a number of important benefits to using a Monorepo (with a Smart Build System).</p> <ul> <li><strong>Consistency</strong> - It's possible to share UI modules, documentation and other such shared packages</li> <li><strong>Visibility</strong> - Everyone in the organisation can see all the code, improving cross-team collaborations and communication</li> <li><strong>Build and Development Caching</strong> - Some tools like NX and Turborepo allow for efficient rebuilds to allow rebuilding only changed packages</li> <li><strong>Single source of truth</strong> - As all changes to the repo are atomic, there won't be a situation where one team is working with outdated legacy code</li> <li><strong>A single CI/CD pipeline</strong> - There is no need for multiple pipelines in each application as we can handle it in one centralized place</li> <li><strong>Deduplication of node modules</strong> - Node modules are the computational equivalent of a black hole already, so any deduplication or help here is always appreciated</li> </ul> <h2>Downsides of a Monorepo</h2> <p>There aren't many downsides to using a Monorepo, but there are some that are worth mentioning:</p> <ul> <li><strong>Slower IDE Performance</strong> - For large Monorepos it's possible to have a slower IDE performance due to the increased number of files, this can slow down imports, intellisense and other IDE features</li> <li><strong>Learning Curve</strong> - Some tools to manage a Monorepo add technical complexity, especially during the setup</li> </ul> <hr /> <h1>How do I make a Monorepo?</h1> <p>Making a Monorepo for a fullstack JavaScript code base is easier than ever, the easiest solution with no additional dependencies is <a href="https://docs.npmjs.com/cli/v7/using-npm/workspaces">NPM Workspaces</a>, and the other two solutions we will cover are <a href="https://nx.dev/">NX</a> and <a href="https://Turborepo.org/">TurboRepo</a>.</p> <h2>Workspaces - Keep It Simple Stupid</h2> <p>Workspace is a generic term that refers to a set of features in the NPM cli which adds support for managing multiple packages from the root of the repository, NPM workspaces allow you to deduplicate node modules and run commands against multiple projects at the same time.</p> <p>Below is an example structure of a repository using NPM workspaces</p> <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">├── package.json └── packages ├── package-a │ └── package.json └── package-b └── package.json</code></pre></div> <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash"><span class="token comment"># ./package.json</span> <span class="token punctuation">{</span> <span class="token punctuation">..</span>. <span class="token string">"workspaces"</span><span class="token builtin class-name">:</span> <span class="token punctuation">[</span><span class="token string">"./packages/*"</span><span class="token punctuation">]</span> <span class="token punctuation">}</span></code></pre></div> <p>Workspaces provides two basic features, as for the most part Workspaces are just code collocation with the ability to run projects from one terminal in conjunction.</p> <ul> <li>Running commands against multiple packages from the top level repository, e.g. <code>npm run test --workspaces</code></li> <li>Deduplication of node modules</li> </ul> <p><strong>Ideal for 💡</strong></p> <ul> <li>Small projects</li> <li>You don't need a smart build system</li> <li>You don't want to add additional dependencies</li> </ul> <h2>NX - Advanced and Powerful</h2> <p>NX is one of the most advanced build systems available and offers some incredibly powerful features to take your codebase to the next level.</p> <p>Some of the most acclaimed features of NX are as follows:</p> <ul> <li>Integrations and plugins for many platforms (including a VSCode Plugin)</li> <li>Rich plugin ecosystem 🚀</li> <li>Ability to build and test only what you need to</li> <li>Automatic project dependency management</li> <li>Vastly faster builds with NX Cloud</li> <li>Distributed task execution</li> <li>Code generation tools</li> </ul> <h3>Create a Next.js NX project</h3> <p>Create a new NX project, with a Next.js application</p> <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">npx create-nx-workspace@latest</code></pre></div> <p>The NX CLI will then ask you which template you would like to use to create this new application, for our case we will choose "Next.js".</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/89b263746955656e62a91a5433dd0f34/50383/nx-install.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 56.75675675675676%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAAsSAAALEgHS3X78AAACRUlEQVQoz0VSyXbaMBRl3UMS4nm2PNuCYJvETAHihAAZ20VPF932///h9kk0p4t79J4k3+HJg4tLE67nQdVdaJoL3QzhBAVcxiW8+OYfJvCT6Rlpg6Do4BOcuJEI+RJGvsLA0EO03RysnSPu7sG6DfL5Fmk5RcZr5LyRa1Z91WfEdO5FBfy4RJBUsreSWwwudA+L/gOz/g3Th1fM335hdfxBDjiUIIRux9DsCLobw/RSWEEOK8xhh5QiqmCzEoafw/QzWPEMg0vVRUPuTDtEGBfY719wOH1g+3jEevuETX/A/cMzVlR36x7jdolq2qGY3KGqF0j5DCyvaSQcGqsxGKoeRdxgRA50VmC6fyW8yNjN4R314wmz4wdS6q+MACOC4sRQ3RQKuVedBBrVpnCezSmy4mK53Uv1MBtjd3hDf/rELfX96Tv179gdP7F9prEsdkjHHcyQE0FJ6zmu7mWwIw42fSaH1w6KcYvy5g42qYiVNwskVYt8IuolynpJ6wq8XaNq7mGxMREJsgIauRWkTjSGU26IUPGgWCGuzQAqDV91/scQsTQ3g+aRCyIw2QRGUBGIjPY0uqfaTBIaXoKAE+GVQa/oRPRhcn6t8Kwso4iL9E8KApOI7OiG3E3kuU5CCpkQDgVclmOx/43ByCYlUhFzEIM16PlFrZM7ASkiHPlfe0KokM4EoSBXLCb/03hCDvWEhpy0sAlePifbW/jlGm6xgs938PgDoSc8gd3+RLb/A53ufhsqGI5sXOoMw2tbpknqHn8Bl4U9KpkNplMAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="nx-install" title="nx-install" src="/static/89b263746955656e62a91a5433dd0f34/50383/nx-install.png" srcset="/static/89b263746955656e62a91a5433dd0f34/1d79a/nx-install.png 185w, /static/89b263746955656e62a91a5433dd0f34/1efb2/nx-install.png 370w, /static/89b263746955656e62a91a5433dd0f34/50383/nx-install.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h3>Create a shared React library</h3> <p>One common usage of NX is to create a shared library which can be used by multiple apps. Shared libraries are created in the <code>libs</code> directory by default.</p> <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">nx g @nrwl/react:lib ui-shared</code></pre></div> <p>Running this command generates a new <code>ui-shared</code> library, which can be used by multiple apps. The beauty of this pattern is that if you change <code>ui-shared</code>, it will rebuild other apps that use it, whereas, if you make a change in <code>next-app</code> NX won't rebuild <code>ui-shared</code>.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/602940f9b0361cd5bfb48bd0935cdc5c/50383/nx-create-react-lib.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 83.78378378378379%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAARCAYAAADdRIy+AAAACXBIWXMAABYlAAAWJQFJUiTwAAACeUlEQVQ4y52USW/bMBCF9WPS2FopkZKofd8cI6mLpCiKFEGAXnrvIYei/euvQzp27SBF0RweKHH5ODNvJMMTPZrxFkIuWFkFVnb5rAJr0qU/gMkRUTEgkD3cYIQXXmn5kdKGtBxl8HhGtbnHeP0JSUWTcQ+eDBDpCBZ2cPwWLqd3QRC/A4sqBPkIXkwIU3VBi3dmistnGQFFxssdTK+ixQa234BR1FFG8/FEwAamW4EFg5YvRgSREj2THFYSKKPslHIYjPeYps869LVT7VN1yqMUzPIahOEWgtKMo2tEyQ1k+R4yv6FS9bCDjkQjq2GYdCirbsAlpcRHAtRH0GFUigiYyR0Yo6gCilws+iKTauwldDbZwhMNAWlzlrTgVFCHFs9BfyI0vRolXSqzLWUz6nKE6QY8HGHTfpXumsw0HL9Gt2whq4lMaM6is2nN8moNtFmDgMrjkkmmt59Xc0qHLJSMlVWRYwNk0dMNNdT72qZFp37evB89MsfyWljuHqLcf1lrNRoXZgxb5IgG6qkkB89LBBml4CUkibVbaFDAJ3CqW8BnqFYTyXJWnsNojNk3LPUT5u0vPN79wMPHn7idn7CtvmPXPyESO1zamT6wclQHFLoTlA6gU/OMRNwhL7+ivyJw/4hCfkEW3iOP7tFkD2D+rEGndTroFHQEXphS2531PWRLpngpLiypZfqpTtl0Xj/8GtRY0eaQFYjTTjvtiVY3uFo8fCXHQ3+BvnA5Ays/0Pd8jWZZqHVa/XNQECeo3wLMCbhD0m1QDBNE1h8L7b4ZGM2Iqw1i6sco76GaXTc2q/4f6JIh3TgjCBeq36i/gDOAivbk+V/A3/kfKaGZF2cUAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="nx-create-react-lib" title="nx-create-react-lib" src="/static/602940f9b0361cd5bfb48bd0935cdc5c/50383/nx-create-react-lib.png" srcset="/static/602940f9b0361cd5bfb48bd0935cdc5c/1d79a/nx-create-react-lib.png 185w, /static/602940f9b0361cd5bfb48bd0935cdc5c/1efb2/nx-create-react-lib.png 370w, /static/602940f9b0361cd5bfb48bd0935cdc5c/50383/nx-create-react-lib.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>Here you can see the process of importing our new shared library, and using it in our Next.js app.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/29124ba6f110a814c416613d915140a7/50383/nx-create-react-lib-use-component.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 55.13513513513514%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAABYlAAAWJQFJUiTwAAABvUlEQVQoz31S226jMBTkZ7aBGIzBQBJuJhAgFzXVbqWq+/8fMjt2QpVG1T6MDPbxnJk59qQe0B3eobdHrETj4IctRNw5+HpCvp2g8wEynSCzM6L8ApWfobIT4uz4BZWf4CXZCHP8i+n8gXwzQume64TavGJTnREmA/cGVOWIphlQNhM27eRqpD7w3CBKO6ylcUK8pJih2z+IVAeV9ogJpfeQSYcgalnUcM+gYPe86LFjk4KKMyrOygHptofmGrHe1npxesBx/kSxm10HZzu8QcjWkUa0vimoOL/ym6o0rWo2E/VDTI2r9VbcbMoJnZmo0MD+2wOLMDZuLQraUj3a4oRC7RHSzepO8Ain0BJk+Yhdc2XIM9bxnofm3vF2KVQGgkQ66RE9XA7Cm7JHeIIqRDIir95Q1RdOa3aDECS2eQYM29ZIEgrZu/1Q3Zo+K3SWg7DGevsKVb5xGAf4y6E0d3V2QHwyJIsJKdmIL2ElSvzyt3gJdt/g+UGJcPpEWv92OS12Ftj/tbTDqvCyrrBaICqX9zM8nwoFJyf1zIc7PuTXfiP/yd6Plu20VMLANxdkuyvD778IluCfVf8P/wCw2mN4fKx4cAAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="nx-create-react-lib-use-component" title="nx-create-react-lib-use-component" src="/static/29124ba6f110a814c416613d915140a7/50383/nx-create-react-lib-use-component.png" srcset="/static/29124ba6f110a814c416613d915140a7/1d79a/nx-create-react-lib-use-component.png 185w, /static/29124ba6f110a814c416613d915140a7/1efb2/nx-create-react-lib-use-component.png 370w, /static/29124ba6f110a814c416613d915140a7/50383/nx-create-react-lib-use-component.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h3>Affected Builds</h3> <p>When you run <code>nx test next-app</code>, you are telling Nx to run the <code>next-app:test</code> task plus all the tasks it depends on.</p> <p>For a small project this is okay, however for a large project you really only want to run the tests for the files and changes you've made to the app. To handle this, NX has the <code>affected</code> command. <code>nx affected --target=next-app</code> will figure out which tasks to run, and only run those tasks.</p> <p>To visualize these "affected libraries", use the <code>nx affected:graph</code> command, in this example below we've added a <code>second-shared</code> lib which is also used in <code>next-app</code>.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/b2442527adc29a63bb29ca1d3d40e898/50383/nx-affected-1.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 61.62162162162163%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAABYlAAAWJQFJUiTwAAAB00lEQVQoz3VSW3KjMBDkMgkYJCOEeIPBxvErdmqr9i/3v0dvjxy7yO7mo63xoOnpnlEQ2jeU/Q3t+IF6uEFnO+j8CO1OSHmmPE1xRkbImboj1vnBQ77n9QVZeULV3WDLM4KV3iBZb5FaFucnWMfC8gJTX0nCnJ2h+F2nOygzI0m3iNTwRJj0UMypbEaoNggiIWTCuQvy/IyiuMI1V3b8YHeSuy1iqo5ZkJgRK91DalYLxOuRmHwc3BMTKhI1xDrb0/IJa1rVjJWlveoMXUpuRiyEC4VLeEJhTwwl809byHw4w3SEyWeqm5HZHaJVi9eoQrhqECUdwrj1p+CVuTDunmq/LE/QtBPr8Z+Of9v7CU+F8qM4cMuNWb+UI2d4huP8RP3j4qMoUvcmPzUKnkoWhd8J7ptc3lvGYbJwRHwp3MAavj098WmMXpmSuXIUMo6yfffxneheKHHKOQvk7WbFm78TiF1d/YKZPpF2v5GX3DQv2WYPWx/gyj2XcyDRhJdk4y0/1ElTIZOmEouQIObFcaCK4QDTyBx3fI8DXNHD2A0K1yPLGqSmQ102XNzwjXRpOfKW1YTrmGGqHEpboXJEUaN2NYq88XFbNYxb5LZdEA7/XcofvwOCrbbDh10AAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="nx-affected-1" title="nx-affected-1" src="/static/b2442527adc29a63bb29ca1d3d40e898/50383/nx-affected-1.png" srcset="/static/b2442527adc29a63bb29ca1d3d40e898/1d79a/nx-affected-1.png 185w, /static/b2442527adc29a63bb29ca1d3d40e898/1efb2/nx-affected-1.png 370w, /static/b2442527adc29a63bb29ca1d3d40e898/50383/nx-affected-1.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/3469701bdeeaf7c5aecccb0ed507944b/50383/nx-affected-1-graph.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 61.08108108108108%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAAAsSAAALEgHS3X78AAABZklEQVQoz4WTW07DMBBFuw/WgMQfS0CsgA82AGIdbIXdsIB+pKJtmrRpE8d52k7Sy4xJWtMmMNJIfuV47vVkdvf4ituHF9zcP+Pp7R0cXddRHnE8nrNzk/bdPTdnc28FbxlgvljD+/InD1/GJLCqKmRZjqquoZVC27ZXh+paQcoMks5xCiGhjRkH8sFDLFCUFfI8vwJSLSiKEtH+gF20pzzYsdL6x55LYNbf2jQNuNpLIFvAoQkQx/H/kkUqEe72VvIYcMiSFMSJsJVpbaaBiRDwN1vrC0ubAjKM5a78wCoaXn7kUagyMl0pTcBiFGjIDl5n+Vzdn20jpSQ5pZ24koePeK7tA/Rw01i/XY9/S06SE5D9YQAHS/Q3IYJthCCMTlIZwPYE4Q7L9QaJSM8/AwPjOLFSeWLa5nQjV1XTQ/Elyo6Vlc77hnpwWHP7EQxM09RKHeS5EvoG6RNX/TmscRRG4WPxiW+iB6QSj5ShIQAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="nx-affected-1-graph" title="nx-affected-1-graph" src="/static/3469701bdeeaf7c5aecccb0ed507944b/50383/nx-affected-1-graph.png" srcset="/static/3469701bdeeaf7c5aecccb0ed507944b/1d79a/nx-affected-1-graph.png 185w, /static/3469701bdeeaf7c5aecccb0ed507944b/1efb2/nx-affected-1-graph.png 370w, /static/3469701bdeeaf7c5aecccb0ed507944b/50383/nx-affected-1-graph.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>Now we're going to make a change to <code>second-shared</code>, as this is used by <code>next-app</code>, and <code>next-app-e2e</code> you should expect that they should both be re-built.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/5060e7b02fdde7f06053151c3fd8f667/50383/nx-affected-2.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 62.70270270270271%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAAB80lEQVQ4y3VS2XKjMBDkZ1LGgJA4hDgExjY2duzEW3nI//9Jb0sOldi7eWiG0sz09BxB1RxR1xMMIfMtRH6EKM+Q5QlKnz0y/YqsIrR7nyGL2Vtj31GYVw/TvTPmjCBRe5+Q6wtBy0fV3nyQKicIuUUq93BxiZqwFgPCpMeacDaMexY4IM72COkLEibo8oKCqqrqDWX7hrq/QbdXViRJsUOU75lAm9o7yRcW8kRtEcnR/wdROqIxNxiSximTlKs2w72v5Q6JnokTRDVDZIMneYYjWlGpyEYEggrCqIVtOAdLpeaAqj2htldUZma7GyTpgCixWDEujBd03r6E9ZfiAbHakNAtgm0b4xZwQMr2ZDE9QJUHuNGEyWPLd9iH9oOYrQkyuwpuwEvgKrYPWJJ+w+LnUnbI8hlFcUbOU9FsXbec59eQnxfw/P8cE7hPJMa7QvEvgSOO0o0/k/WPc3H+u2/w/tjFOMKUN2TMB1R2pEIetdrxaCeezIFb28Fu/qA0J7xE1hf9LrRhN2c/84wzLszsiYNyINn0CTV+kPgKk48k4G22RDOh70bkBU9CWqroH0iX2f78Dy7bElp3VNcjVR3n2UAXDaTkW9bwPmuq7zA0BhntyhP+vqDg86jQmdonWtq2alBrEulvW5WNJ1rIFpX/w18ifaRvVJxxAQAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="nx-affected-2" title="nx-affected-2" src="/static/5060e7b02fdde7f06053151c3fd8f667/50383/nx-affected-2.png" srcset="/static/5060e7b02fdde7f06053151c3fd8f667/1d79a/nx-affected-2.png 185w, /static/5060e7b02fdde7f06053151c3fd8f667/1efb2/nx-affected-2.png 370w, /static/5060e7b02fdde7f06053151c3fd8f667/50383/nx-affected-2.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/3ad6e7491021139bd74a571cb9f7919b/50383/nx-affected-2-graph.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 61.08108108108108%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAAAsSAAALEgHS3X78AAAB80lEQVQoz3VTwW7UMBDNh/ARHDjwCxz4Ay6oHACpRcAfAEJIW8EnFFXQQy8ItmqRekCAKjhw6gEhkKqq23bb3TSbJXHseGw/xpNttdkuVkaJPfbzmzcvybWbi7h64z6uXL+F2487iMN5Dx8Cwnnw2uWYyk9FcnB0ihj7vRP0jk7gGay1KYJXGr6smlDNO5CbD2iMQZ6PUVUVjNZwbrLReWHij4cwj16hXlpG/fAlzINlmLvPQdvfG6auDZyUfGOa5VDaoChKZugumAlgXoDWt6Ffd6HXtkBrH2FXN+B+7k0A/QxDBlFnGUgpVBxu+kYfRMs4hsEig5PvCx1n5YmAuixRRkAuWV0CbFgGLlutvIeKDN9sgr7tzmUngEWp0B+kKFn4qGNLQyI5SLu/YRaeiIb63gvUK91GjjmNSUbckN5hXxojGkbAGYtYXvOFAo3GsMMMoTKNbeaVHD1nLYH4kDAMvInZun4KP8hQHw84hghpjlBb1Jx3UdcwcWMILc8mRfFXtIsT4gM+Mup+gV54CrPYkRL1UgfmzjPQu08gzmdphlOW6ZB9Oxrl0ijvJ4D5eMyAlUystVKi+3MA++EzaHMHtPEVtLUjc/drXy7U7FfF1UT9o49bJUftSk5E2pab4M87+7/wHtNDfkx+lK3xdu8H/gHU745ueaMLxQAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="nx-affected-2-graph" title="nx-affected-2-graph" src="/static/3ad6e7491021139bd74a571cb9f7919b/50383/nx-affected-2-graph.png" srcset="/static/3ad6e7491021139bd74a571cb9f7919b/1d79a/nx-affected-2-graph.png 185w, /static/3ad6e7491021139bd74a571cb9f7919b/1efb2/nx-affected-2-graph.png 370w, /static/3ad6e7491021139bd74a571cb9f7919b/50383/nx-affected-2-graph.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>As epected you can see that the <code>next-app</code> and <code>next-app-e2e</code> projects are affected, but the <code>ui-shared</code> lib is not.</p> <h3>Incremental Builds</h3> <p>Once you've got an established NX Monorepo, you will likely have many applications and libraries. It is common that you will only be editing a small subset of these applications at once, Incremental builds allows NX to only rebuild a small subset of the libraries, which results in much better performance.</p> <p>This then works closely with NX Cloud. With NX Cloud you and your team mates share the same cache, allowing you all to benefit from improved performance.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/9ce6a1a70dc8245c6d67f493842112eb/50383/nx-incremental.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 62.16216216216216%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAAAsSAAALEgHS3X78AAAB+klEQVQoz32Ty28TMRCHwx/AX8UBqVSiFAKIAm0lLkg5cOTAHSE1F1Rx41XEgQPideARkhS48Ggr0S2iFQdUECDIBqI0ye56bX+M7TRSaIMl745ndj7/xp4tICNJEvI8x1oDMq3OZa1JMnmnKW/Wu8RdLbGMVNZaa4wxO6a1loIDOsNNYyy9TN5sD+uCnLpmeXnhMqyvOI/saQY5/04PVCqXp6HZsZy70eHr2zXotT3YivKZW1DddwQq94JP5/8HplnmMvnVgeKlBquTR7HRkk82KmNmAWrjp6H+CO18ssnIkoMRCoy7MFWOiQ4V4cNKcDqFAqweOCnAh75kyWbUKLRaLdrtLTE1DVF4Yk6AotBEy0GhqJ92wLEpePG8r1qNVqh80B+1V+iBQwqVB9bGZ7F3z5PrTfo3s7tC98gEOqywKGf4LuRJbPqmpXawhC7vIfs25/1aK98+7oKMkTbaecsB6M/w8DFpkU8+0aiUWadwogTze9E/5/ttpbYbK3wnUA/0/WdDyAEnLzZ4PzZB/rgkVX1x3cTxq4bK/jNQLpD/KHtQdUMTLTwjv38W3bwT4FYHhe4vUVlKJ1G8/rjF56cV2mvXyXubJKmmHv1m48kizeXbtONIvjfUVpssPXhFvHiFP9/r0nrhGLzC3YYdskKRZuC3A2vYB38Bxi53HzwrEa0AAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="nx-incremental-builds" title="nx-incremental-builds" src="/static/9ce6a1a70dc8245c6d67f493842112eb/50383/nx-incremental.png" srcset="/static/9ce6a1a70dc8245c6d67f493842112eb/1d79a/nx-incremental.png 185w, /static/9ce6a1a70dc8245c6d67f493842112eb/1efb2/nx-incremental.png 370w, /static/9ce6a1a70dc8245c6d67f493842112eb/50383/nx-incremental.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>The above chart is <a href="https://nx.dev/ci/incremental-builds">from NX</a> and highlights the performance differences between normal builds, and incremental builds (both cold and warm).</p> <blockquote> <p>Running the Nx incremental build having already cached results from a previous run or from some other coworker that executed the build before. In a real world scenario, we expect always some kind of cached results either of the entire workspace or part of it. This is where the teams really get the value and speed improvements.</p> </blockquote> <p><strong>Ideal for 💡</strong></p> <ul> <li>Medium to massive projects</li> <li>You want <em>all</em> of the fancy modern features you can get</li> <li>You don't mind dealing with some complicated looking <code>nx.json</code> files and tuning them to your liking</li> <li>You want a battle hardened, community supported platform with industry backing</li> </ul> <h2>Turborepo - The New Kid on the Block</h2> <p>Turborepo is a recently open sourced project acquired by Vercel, the company which brought us Next.js.</p> <p>Turborepo's goal is to take what's great about other build systems such as Lerna, and NX, whilst shipping it in a small simple package, which works hard to stay out of your way.</p> <h3>Setup</h3> <p>Setting up Turborepo is as easy as it gets, just run <code>npx create-turbo@latest</code>.</p> <p>This sets up an example project, with a <code>web</code> and <code>docs</code> apps, and a shared <code>ui</code> package. Unlike NX there is a lot less boilerplate and this resembles a typical project structure you'd see with a "normal" Monorepo, this makes Turborepo much more approachable to a monorepo newcomer.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/1f6cde560aa0061c2ab7702db372c790/50383/turbo-structure.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 66.48648648648648%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAAB30lEQVQ4y4VT2XLbMAzU11T3fcuSLJE6fClxE3s60///jy3A2rWVpO0DBiAIrnaXorbpTpD7H/AiCcvbwo8HqgXcbI+sPiFMRwTFAVG2o/5IMd3iUYfpDiHtp9UJWkGHanEhoElFkExwCdByt4jjDkXeIS8F8qyH7Tf4ZpbQ7QoGhW5VqtatEqZTw3BaaD4xi6tX2thQo1lFSuyydIHrS/ghf1DAdOtPc+Y9uwRohQJhvVCzVQ3jadN0O9Q5Sc93xLh96v89iGGPuj2QzJkkdcpHDt7kdRRKuEEP4wnwI0NjxdDr4OcviIsjuH4Acm6h06B+k/UAYzZfs9VMYhFvFrrFI2xm4jx79G95X1mg2SQ5as509XuS2P9XFodu16u5+5pD8xKJjBjavlgNcXaoZ3tCSb97yr9TEI/KFsNuVM+li3VIHXut2UGn/POTeQXIA0W1IKEfPClmsmRClPJPPCEtDwpIzVFWD4EBQwI0qQjyI7xw+APIElIC6cQZcr6g6V7RUh3ns3o51m3mo1wl2Sa6Hkl4lspDebnHMFwhxgvkdMEwX9HJt995eMOmXdAP72odE3s+oy6l6c+YTz+VTyuG5YS2f0FJT1PSoTsoZzldsRXfVS3Gd1TNoizi878AMLioFRZysdcAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="turbo-structure" title="turbo-structure" src="/static/1f6cde560aa0061c2ab7702db372c790/50383/turbo-structure.png" srcset="/static/1f6cde560aa0061c2ab7702db372c790/1d79a/turbo-structure.png 185w, /static/1f6cde560aa0061c2ab7702db372c790/1efb2/turbo-structure.png 370w, /static/1f6cde560aa0061c2ab7702db372c790/50383/turbo-structure.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h3>Affected</h3> <p>Turborepo has its own flavour of <code>affected</code> from NX, both work very similarly, however Turborepo doesn't produce an interactive graph like NX does, Turborepo has a basic graphviz image export.</p> <h3>Other Features</h3> <p>Turborepo has a few other features:</p> <ul> <li>Cloud caching</li> <li>Parallel execution</li> <li>Profile in the browser</li> </ul> <p><strong>Ideal for 💡</strong></p> <ul> <li>Vercel users, Remote Caching works great</li> <li>Small to medium projects</li> <li>You don't mind losing some features of NX</li> </ul> <h2>Feature Comparison</h2> <p>Below is a feature comparison for Workspaces, NX, and Turborepo.</p> <table> <thead> <tr> <th>Feature</th> <th>Workspaces</th> <th>NX</th> <th>Turborepo</th> </tr> </thead> <tbody> <tr> <td>Node Module Deduplication</td> <td>✅</td> <td>✅</td> <td>✅</td> </tr> <tr> <td>Run Commands Across Projects</td> <td>✅</td> <td>✅</td> <td>✅</td> </tr> <tr> <td>Shared Libraries</td> <td>✅</td> <td>✅</td> <td>✅</td> </tr> <tr> <td>Profile in Browser</td> <td>❌</td> <td>❌</td> <td>✅</td> </tr> <tr> <td>Dependency Graph</td> <td>❌</td> <td>✅</td> <td>🚧 (non interactive)</td> </tr> <tr> <td>Incremental Builds</td> <td>❌</td> <td>✅</td> <td>✅</td> </tr> <tr> <td>Cloud Caching</td> <td>❌</td> <td>✅</td> <td>✅</td> </tr> <tr> <td>Code Generation</td> <td>❌</td> <td>✅</td> <td>❌</td> </tr> <tr> <td>Distributed Task Execution</td> <td>❌</td> <td>✅</td> <td>❌</td> </tr> </tbody> </table> <h2>Conclusion</h2> <p>Hopefully you now appreciate the importance of a Smart Build System within a monorepo. As much as I'd recommend giving both NX and Turborepo a shot, if you don't have time to experiment with both, <strong>I recommend trying NX</strong>.</p> <p>Turborepo has most of the features that NX has, but NX does that and more, in a more performant package. Given the current state of Turborepo due to it's recent inception, I'd rather have a NX monorepo that has been battle hardened which you know won't let you down.</p> <p>However, with Turborepo's recent backing of Vercel, I expect NX to face some strong competition in 2022, and as you all know, competition brings innovation.</p> <h2>References and Reading Materials</h2> <ul> <li><a href="https://nx.dev/docs/">NX Documentation</a></li> <li><a href="https://Turborepo.org/docs">Turborepo Docs</a></li> <li><a href="https://nx.dev/guides/turbo-and-nx?utm_source=twitter&#x26;utm_medium=ads&#x26;utm_campaign=dec2021">NX and Turborepo comparison</a> (Disclaimer, it's from NX)</li> <li><a href="https://monorepo.tools/">monorepo.tools</a> Fantastic Resource! 🔥</li> </ul><![CDATA[Bringing the client closer to their product using Contentful]]>/2022/02/client-closer-to-product-using-contentful//2022/02/client-closer-to-product-using-contentful/Tue, 15 Feb 2022 00:00:00 GMT<p>You need to make sure that your website content keeps up-to-date with changing designs and client requirements. The trade-off? This often impacts developer time. Small changes, like adjusting an image or a piece of text, seem like a 10-minute job. In reality, they take up more capacity as they still need to go through the full development processes (at Theodo these include code review, deploying, functional review, validation, etc. to ensure quality and avoid regressions).</p> <h2>Content Management Systems (CMS)</h2> <p>Utilising a CMS means that you can create, manage and modify the content of a website without needing any code changes. A developer has to integrate the CMS using its API, however, once this is done, anyone with the correct permissions can adjust the content. This means that content changes now become much simpler and the client can benefit from up-to-date content, as well as saving developer time over the longer term.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/6f43b0dfeb85466c78232b51615f076d/50383/cms-made-simple.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 75.13513513513513%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAPCAYAAADkmO9VAAAACXBIWXMAAAsSAAALEgHS3X78AAACoElEQVQ4y2NgAIKE+ukMMDqhbjqErp8BpyFyEJooEF87FWjQNIa0jqVY5ZEtJGwY0KDU1nlwfmHPUpb6dWfZQrqWcSfWTedNaJjJhGyYoUYshNaMZWAtWQJmswBpFiibIaVlLtTg6bypzXNEfAs6hWw9oiVqUspFopvmcCXWTxdOqp3GCFKzd8MxprOTNrIcPHSZCeYA9uIFjNpNK1hk61YyIXuJM7l2Ch+IXVVWrldZmLstPb3UAyxXN5Unvn4GN7rPHvQtZGCIaEYRM2zfxMAQV9sLCnCxuXM6mUGC8+a0mc2av+KWnl8V2MD0xumcaTXThUDso6sOeJ1vWdJ9dN85G5ghzEWL5MWrl08SrV9jCRZIrJ/HmtwwlR/Ebur6nNzS+ToOxD5dNFXn/7w2AbChZX3iIHrb/ivn351+/3/XmTs7YQbyl87vrDt09b9h3cI9YIHMpi2s6V3JvG6J/+RXrfvwv7P3yn8Ghmu8F5dnrSnqnruFoWStYHXbdB6whVuu+ecuu7xt0eFbbjADeUvnG4lmT93HFt0dDBbIaVnDGFnZL9C79CxnVuH29pDo9bX//zMw326Srp2WGFmi0XdMEKSuHiiIDJJXnUfmgnzCCOelty5lKV+wmRfEPgPTuP0/+38kQzoZRBma995jmnb6CVPX7tvgGP3x4T3jovOPmduPP2diSJgNMRBJk4Canrm8oa2XlLlHqJyWhrIYJwMryBI2mILlW48wPDj3GMz+9OUriovvPnqK4Miq64MoULIBhZUEECuDLGBmZuFiZGICG8jMCjH34bPXDH///gWzn756p/T4xRuLJy/fSoP4T168AaZyNnYGUgCagZJAAzWBBorADSQXfPn2HYX/9OVb8gxCcyED0IWMT6CGgVwIAB62G0MPqISwAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Simple diagram showing CMS process" title="Simple diagram showing CMS process" src="/static/6f43b0dfeb85466c78232b51615f076d/50383/cms-made-simple.png" srcset="/static/6f43b0dfeb85466c78232b51615f076d/1d79a/cms-made-simple.png 185w, /static/6f43b0dfeb85466c78232b51615f076d/1efb2/cms-made-simple.png 370w, /static/6f43b0dfeb85466c78232b51615f076d/50383/cms-made-simple.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h2>Introducing Contentful</h2> <p>Contentful goes beyond CMS; not only does it allow you to plug adjustable content into your website, it also incorporates additional features to optimise your content.</p> <p>As a quick introduction, Contentful is a content management platform with a very comprehensible Content Delivery API. It is available in the most popular programming languages (e.g. Javascript, Python), a great help when you are working on a variety of projects utilising different stacks. Overall, some of the best advantages include the ease of setup, clear and practical JavaScript SDK, and Contentful’s intuitive platform.</p> <p>Before delving into the advantages and disadvantages of Contentful, let’s introduce the technical side: how to integrate it into your code.</p> <h2>Code example</h2> <h3>Setting up</h3> <p>The first, and essential, part of using Contentful is to create an account on their platform <a href="https://www.contentful.com/">https://www.contentful.com/</a>. Assuming that this has all been done, we can now turn our eye to integrating it with our code.</p> <p>If you don’t yet have Contentful installed and are using Node.js, you just need to install the package using:</p> <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">npm install contentful</code></pre></div> <p>Next, you need to initialise the client. To do this, you will need an API key and space ID (some quick steps can be found <a href="https://www.contentful.com/r/knowledgebase/creating-a-website-in-five-minutes/">here</a>). Although optional, you can also specify an environment. We’ll talk about this later, but it is another key advantage of using Contentful.</p> <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">var client = contentful.createClient({ space: &#39;&lt;space_id&gt;&#39;, accessToken: &#39;&lt;access_token&gt;&#39;, environment: &#39;&lt;environment_name&gt;&#39;, });</code></pre></div> <p>That’s all you need! Once Contentful has been set up, you can now use it throughout your project.</p> <h3>Fetching content</h3> <p>There are lots of ways of retrieving content. You can use a specific entry id to retrieve a specific item, however, you can also retrieve groups of content. One of the most useful tools to retrieve content is being able to retrieve by content type. Within the Contentful platform, you can create ‘content types' that define the types of data that you will be adding to your website. For example, you might have a simple type such as <code>colour</code> which will just have a short piece of text (i.e. a title) or you might have a more complicated type such as <code>author</code> which will require a name, a description, an image, etc. To fetch this content type (e.g. fetch all the <code>author</code> types), you need to call:</p> <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">const authorResponse = await client.getEntries&lt;Author&gt;({ content_type: &#39;author&#39;, })</code></pre></div> <p>Where <code>Author</code>, in this case, is the type defined by Contentful which needs to be imported. The response can then be used anywhere you need. Within Typescript, the <code>Author</code> type would need to be defined. Although this can be done manually, it is more efficient to use an automatic type generator such as <a href="https://github.com/contentful-userland/cf-content-types-generator">this one</a>.</p> <h3>Optimisation</h3> <p>Contentful can also be used to adjust and optimise your content. A key example is with images. There are many options available (see <a href="https://www.contentful.com/developers/docs/references/images-api/">documentation</a>), however one of the most interesting ones I came across was the ability to change the quality of the image. By reducing the quality of the image, the content can be rendered more quickly on your page, avoiding delays. Once you have retrieved the image URL from Contentful, all you need is to adjust it and specify the quality:</p> <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">const imageUrlWithAdjustedQuality = `${imageUrl}?fm=jpg&amp;q=${quality}`</code></pre></div> <p>With Contentful, the quality is a percentage and can be set to be between 1 and 100.</p> <p>These are just two examples of what you can do with Contentful. I would highly recommend exploring their documentation and having a look at implementing it yourself too.</p> <h3>Environments</h3> <p>Earlier, I mentioned that we can set the environment when initialising the client. Contentful has a wonderful feature where you can have content in different environments. This means that changes made in Contentful don’t need to directly impact your production website. Contentful does this in a very similar way to Github where you can have different branches (e.g. <code>master</code>, <code>staging</code>), create and destroy branches. The key process to releasing a new branch is as follows:</p> <ol> <li>Create a new branch (copied off of the master branch) where you make all your content changes</li> <li>Once the changes are complete, point <code>master</code> at this new branch</li> <li>You can either delete the old <code>master</code> branch or you can keep it in order to revert to an older version if needed</li> </ol> <p>Although this can be done through the terminal, it is highly recommended to use the Contentful platform as it is quite straightforward and can avoid unwelcome surprises.</p> <h2>Key advantages of using Contentful</h2> <p>A key feature that singles out Contentful compared to other CMS is the ability to manipulate images. It does so by not only adjusting the format and size of your image, but also by enabling cropping and adjusting the quality you would like to have - a great optimiser of loading speed.</p> <p>While CMS is often thought of as being able to integrate content such as images, videos and text, Contentful takes it a step further. While the general page layout is predetermined, its content can be completely set through Contentful. This is because text can be done in markdown, giving more flexibility to what can be displayed.</p> <p>In the code example, we had a look at fetching all items within a content type. If all the data is useful to you, then this is great. However, you may only want to fetch a specific group of items. For example, if you have a <code>Book</code> content type, you may only want to fetch items with the field <code>genre</code> corresponding to <code>'Mystery'</code>. This is done using <a href="https://www.contentful.com/developers/docs/concepts/relational-queries/">relational queries</a>, e.g. using the API call:</p> <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">https://cdn.contentful.com/spaces/my-space-id/entries/ ?content_type=books &amp;fields.genre.sys.contentType.sys.id=genre &amp;fields.genre.fields.label[all]=Mystery</code></pre></div> <p>However, the key benefit of using Contentful compared to other CMS is the simplicity of integration.</p> <h2>Disadvantages and a word of warning</h2> <h3>Speed</h3> <p>Fetching all your content from Contentful seems like a dream come true, no more hardcoding text and images, freeing up developer time and making sure that your content is up-to-date. However, this does come with a warning. Fetching all the content whenever a page is loaded can be very slow. The more data you fetch from Contentful, the longer your loading time. However, this does not necessarily need to be an issue depending on which framework you are using. Contentful used alongside NextJS is one of the best solutions. NextJS uses static site generation (SSG), meaning that your content does not need to be refetched whenever you load a page - a significant time-saver. However, this does mean that if you make changes to Contentful, you will need to rebuild the site in order to see these changes. Rather than SSG, you can also look into using Incremental Static Regeneration (ISR) which uses static-generation per page rather than the entire site and would avoid the need to rebuild the entire site when a change is made on Contentful.</p> <h3>Field type</h3> <p>When fetching content types, there are also limitations on the field type we can filter on. One of the types that it is not possible to filter by is <code>Array&#x3C;Link></code>. A further note to consider about the <code>Link</code> type is the issue of depth. Links are a great way to model the relationship between content items, however as the depth increases, this can cascade into a circular relationship. By default, you will only access the top level, however, if you do need a higher depth, it is important to understand the possible consequences of expanding the maximum depth.</p> <h3>Other points to consider</h3> <p>Another point to consider is that Contentful cannot be installed locally for local development. To work on your project with your content fetched from Contentful, you will need to have an internet connection. However, as this is also the case for other CMS, it is not a strong disadvantage. One of its key challenges is its cost. At about $489/month for a team, Contentful can be quite costly, especially for smaller projects.</p> <h2>Conclusion</h2> <p>Overall, Contentful is a great way to optimise time for developers and ensure that the client has control of their content. However, you have to be careful about when to use this as projects with a low budget and with small amounts of content might not need such a powerful tool. Another key consideration is whether you are using NextJS on your project. If you are not, you might want to think about switching over to make sure that your content is loaded quickly and efficiently.</p> <p>More documentation: <a href="https://www.contentful.com/developers/docs/references/content-delivery-api/">https://www.contentful.com/developers/docs/references/content-delivery-api/</a></p><![CDATA[Logging uWSGI errors and alarms to Sentry]]>/2022/02/logging-uwsgi-alarms-errors-sentry//2022/02/logging-uwsgi-alarms-errors-sentry/Wed, 02 Feb 2022 00:00:00 GMT<p>If you already use Sentry to manage your application errors and uWSGI as a WSGI server for your Python project (such as Flask or Django), <strong>you are one step away to improve your monitoring setup in a matter of minutes</strong>.</p> <p>It is important to do so: even if you have Sentry configured for your app, <strong>if the WSGI layer that is running it fails you won't get any Sentry issue about it</strong>.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/9a5ff88cc9cba305d5e5754a1fff57b8/50383/uwsgi-architecture.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 19.459459459459456%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAECAYAAACOXx+WAAAACXBIWXMAAAsSAAALEgHS3X78AAAAw0lEQVQY042OCQ6DMAwE+f/joEgVN5QrISAQhLsc3SbuB2rJirSxx2MwxqC7aRqs64q6rhEEAeI4RhiG6Pse0zQhz3NwziGEwDAMWJYFXddRSylp7jxPGEVR0KL+aNsWaZrC8zyYpgnbtmmZKRBXh1zXJfA4jiThRxFlej5Jkh9wnmcy0O993xTu+45t28jifRzAdWFUh2MF0PZZllEuWQlRVSSj7fUhA3/URwEVHVLZlmVJAlSei5fv4+k4eFgWKgX/Ag5SLLvgIj9kAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="The layers of a standard Python web server: an HTTP server with a WSGI server and a Python application" title="The layers of a standard Python web server: an HTTP server with a WSGI server and a Python application" src="/static/9a5ff88cc9cba305d5e5754a1fff57b8/50383/uwsgi-architecture.png" srcset="/static/9a5ff88cc9cba305d5e5754a1fff57b8/1d79a/uwsgi-architecture.png 185w, /static/9a5ff88cc9cba305d5e5754a1fff57b8/1efb2/uwsgi-architecture.png 370w, /static/9a5ff88cc9cba305d5e5754a1fff57b8/50383/uwsgi-architecture.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>With uWSGI <strong>there are some events that you want to watch for</strong>, and triggering a Sentry issue is one easy way of fitting this into your existing error management flow.</p> <p>Such events include:</p> <ul> <li>when <strong>a uWSGI worker is killed</strong> because it is serving a request for an excessive long time (<a href="https://uwsgi-docs.readthedocs.io/en/latest/Glossary.html">the "harakiri" behavior</a>)</li> <li>when <strong>the listen queue of the uWSGI socket is full</strong></li> <li>when <strong>a segfault happens</strong></li> <li>or any non-applicative log line that you want to monitor</li> </ul> <h2>How to integrate uWSGI with Sentry</h2> <p>Unbit (the company behind uWSGI) wrote <a href="https://github.com/unbit/uwsgi-sentry">a plugin exactly for this use case</a>!</p> <p>Make sure that a libcurl package is installed on the machine that runs uWSGI, for instance:</p> <p><code>apt-get install libcurl4-openssl-dev</code></p> <p>Install the plugin before running uWSGI:</p> <p><code>uwsgi --build-plugin https://github.com/theodo/uwsgi-sentry</code></p> <blockquote> <p><em>NB: this is a fork of the original plugin that makes it work for current versions of uWSGI and Sentry, I will update this article if it is merged into the main repo.</em></p> </blockquote> <p>Configure the plugin in your uWSGI INI file:</p> <div class="gatsby-highlight" data-language="ini"><pre class="language-ini"><code class="language-ini"><span class="token selector">[uwsgi]</span> <span class="token constant">plugin</span> <span class="token attr-value"><span class="token punctuation">=</span> sentry</span> <span class="token comment">; Defines what to do when the "logsentry" alarm is triggered</span> <span class="token constant">alarm</span> <span class="token attr-value"><span class="token punctuation">=</span> logsentry sentry:dsn=https://your@project.sentry.io/dsn,logger=uwsgi.sentry</span> <span class="token comment">; Alarm for listen queue full</span> <span class="token constant">alarm-backlog</span> <span class="token attr-value"><span class="token punctuation">=</span> logsentry</span> <span class="token comment">; Alarm for segfault</span> <span class="token constant">alarm-segfault</span> <span class="token attr-value"><span class="token punctuation">=</span> logsentry</span> <span class="token comment">; Alarm for harakiri (alarm-log uses regexp to match log lines)</span> <span class="token constant">alarm-log</span> <span class="token attr-value"><span class="token punctuation">=</span> logsentry HARAKIRI ON WORKER</span></code></pre></div> <p>You can find more customization options in the <a href="https://uwsgi-docs.readthedocs.io/en/latest/AlarmSubsystem.html">Alarm subystem docs</a>.</p> <h2>Conclusion</h2> <p>Deploy this change to your server and <strong>bam, you're done!</strong></p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/81db8e1165b63381e43a280d4b4b29d0/50383/example-error-in-sentry.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 43.78378378378379%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAIAAAC9o5sfAAAACXBIWXMAAAsSAAALEgHS3X78AAABa0lEQVQoz41Qi3KjMAzk//+tc53mmKS5QpoUAtj4bUt+YHLivuA0tiyvtPJazfXSfV37++2n7+7n9nr+/fnn0rWn8+f5SzAxPsaPX+3728fpvW1Pl+ExSjbOP70Wc8lbswzT4/s53J9yVWJVbOJSKL6sfBEp5uACZ4LNnBDBlbc+BgNWxmAPMt2NcimWlLIznjyhVjvvglYmeECIW6kREzUyhHsMPkKIEXNDFYLLnAoVsWUlTox5mdihYuGU1dqQBGvcyiUVGG2Nss55Aptbfw8OJNOSazYJipUwarWEkCJvwSjPZ0lZ8NEq7wxoYbV0bFLNd0eEiJgAkndIAUIiVcEjwKENMdOVQHoKQ0qYCaSv5Via7jYysy2mTmqbdJ3VRmuUZVjLJMtTFIeVPOZ9r3UrGwQsuazs+GDTD/bJPEHOQTiGgUe61Lq/aO/7i6z+OyrZVmk61MJZh4BNIqWBOAAHEygmMhW9/sP+Aihp/d7AjeIoAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Example of a Harakiri issue raised in Sentry" title="Example of a Harakiri issue raised in Sentry" src="/static/81db8e1165b63381e43a280d4b4b29d0/50383/example-error-in-sentry.png" srcset="/static/81db8e1165b63381e43a280d4b4b29d0/1d79a/example-error-in-sentry.png 185w, /static/81db8e1165b63381e43a280d4b4b29d0/1efb2/example-error-in-sentry.png 370w, /static/81db8e1165b63381e43a280d4b4b29d0/50383/example-error-in-sentry.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>After this, you could reflect back on what other component of your architecture you are missing alerts on.</p> <p><strong>Bonus:</strong> if you are loving Sentry as we do, here are <a href="https://blog.theodo.com/2020/03/use-sentry-effectively/">some tips on how to use Sentry effectively</a>.</p><![CDATA[React Hooks and Tips to Avoid Useless Component Render Applied on Lists]]>/2022/01/react-list-hooks-avoid-render//2022/01/react-list-hooks-avoid-render/Mon, 31 Jan 2022 00:00:00 GMT<style type="text/css"> code.language-tsx{ font-size: 10pt; } img.small{ max-height: 10px; } .column-list { display: flex; justify-content: space-between; } .column { padding: 0 1em; } blockquote { border-left: 3px solid rgb(55, 53, 47); padding-left: 20px; } code{ background-color: rgba(135, 131, 120, 0.15); font-size: 85%; color: #eb5757; } </style> <p>A few weeks ago, I encountered <strong>children list rerender issues</strong> on the project I was working on. In this article you will learn :</p> <ul> <li><strong>how I debugged a react performance issue</strong></li> <li>why <strong>virtualization is not always suitable</strong> for list rendering issues</li> <li>what is <strong>memoization</strong></li> <li>how to <strong>memoize react components and functions</strong> with <strong>react hooks</strong> and <strong>React.memo()</strong> to <strong>prevent a component from re-rendering</strong>.</li> </ul> <h2>Overview on the performance issue</h2> <p>I’m working on a <strong>platform to purchase orders from suppliers</strong>. Each order is composed of clothing items. Here is a typical example of the interface.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/ab96df8d727d529515064ca2070f53ed/50383/website-screen.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 50.810810810810814%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAYAAAC0VX7mAAAACXBIWXMAAAsTAAALEwEAmpwYAAABiklEQVQoz4VS247UMAyd//8aEFoQaKV95wN4QbOznfSWtmmaZtI06eVgZxgYWLS4Ojp24py4dg7rusA7h3maEvzlQuwQyH8LbnRoK4u2tomVtIjzjMNoRpSVhOoHPGdnCIJWCkbrV+D1vutgR015HR7eCXz6IPDxfY7Hz2fI0wsOdugxNBWcUciyE5q2w7puYNvvwBbigjkE/Mt2+vqioArbErIsEN0AcfwGay32fSfR9RW2bUt7zAy+mLFte7pMJUHVoBFHTM1zEtSDSVXcH/wfUgHbCl2RoDEGfd+nskVVUy81Yoy/sCzLHwf/Bq+zPX7p8fXpyENhQZ0WK9nATR7zHODclHj6GYcQMfk5+Z6YfX+LZ0+DvUB8z65T7lQPSwdf8uKaRL/M8PQMWDhSf+LyGzOLsxDxQj1safqKh1uVNGVNw6gaHAuJU15iGAZwG5gHc+ffQHF6PlIm6LZFLQSKPIepaxwWetQTJXgSDpQcxxExsbnyzb/FtO8bCZdlcOKc4EkstE3a+wEPWP/wiqqi4wAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="website screen" title="website screen" src="/static/ab96df8d727d529515064ca2070f53ed/50383/website-screen.png" srcset="/static/ab96df8d727d529515064ca2070f53ed/1d79a/website-screen.png 185w, /static/ab96df8d727d529515064ca2070f53ed/1efb2/website-screen.png 370w, /static/ab96df8d727d529515064ca2070f53ed/50383/website-screen.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>Let’s take a very simple example: I am ordering from my supplier Theodo, producing some top quality shirts. I want to order their famous shirts in three colors (white, red and blue). <strong>On the left there is a sidebar</strong> where you can select a <strong>card</strong>, here there are three cards, one for each color. <strong>On the right there is a form</strong> with the information relative to the selected card.</p> <p>When another card is clicked, the <strong>form values</strong> on the right is now displaying the <strong>information related to the new selected item</strong>.</p> <p>For the new year, a big order has to be done containing more than 600 items. In this situation, the list takes a huge amount of time to load, or doesn’t even load at all. And if I’m lucky enough to have the 600 items displayed inside the sidebar, when another item is clicked on, I’m also waiting... <strong>This performance issue is terrible in terms of user experience</strong> and needs to be fixed!</p> <p><img src="/64e438509a5e6ecdd8b45c7ddcec7565/website.gif" alt="website.gif"></p> <p>First, I need to make sure whether the performance issue actually comes from React. To do so, the user interaction can be recorded with the <a href="https://developer.chrome.com/docs/devtools/evaluate-performance/">Google devTools performance tab</a>. When JavaScript is executed it is accounted in scripting. Here, scripting is taking most of the time. I can investigate more on what is happening.</p> <div style="width: 50%"> <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; "> <a class="gatsby-resp-image-link" href="/static/66e61201f0f0d61019b40124f9789aa0/50383/performance-devtools.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 51.891891891891895%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAYAAAC0VX7mAAAACXBIWXMAAAsTAAALEwEAmpwYAAACQ0lEQVQoz1VSS2gTQRjO1YMg1rDNa5PdPHf2vXlsk6aVphE1RMFWaYn4jHgQFcQiigdJ7KFWURD14kHRiyC9VL0Jgie1N4X25KHgTUQMiBSUz/knGvHw7+zM/HzzPf5AQklBTWYQicrYFgxDCsWRUJJQ1BTiiSTontZkKguNmchkdf6fw/SeMdiWiTTfpzOa6KMK0CcYHEahWMaxIy00d1YQlVX+gCJA6V6Oq8hpBgqlMmwnD2Y4OHuijhE/D9stwbI9RGOJPmAoHEOn28XPX0Dv8xp6qwt49fQ88q71H2hMVjg7BosDZnMMe3edgeOMckACLYgSgNNTM/jxfQN37t6D7/vY16xi/U0XL5+c44DqQArJzuZ0mJaLLGe7u3GAsy3C4HtibVqe6AksP5zD8qN5hCKK8GJIUtHaP4ne2nXsmPCFp4qaHvhpcnlMt9E4XIPLJXueL85IsgB8++wCHt8+Cs3yMVmvw+Ke1Guj+La6iKnmGIKSzENLCx+JGflFTNvtkyhyT4mh4xX5mdEHvHGlhS/vF1HxHWzaLGHLUBQPbh7Hp5V5GIYuAlL+SE6lcwKQ0m4dPAQvXxLe0aoxqw/IGMPKi0v4+uEalu6fwrvnF7Hx8RZOtxsDdtRIlS+OwPVKIvFO5yoq1XEYpismxHYKfcAwT5IxDQuXZ/F6aU6AklQplBDM/gaS4v6SNN10hGRipfPxoUDICppTkTKNBcnaGowNqh8EAf0bGfKqOl6DX67C4TKl4YgA3z5RFw9QD/X+BsmGVql445t+AAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"></span> <img class="gatsby-resp-image-image" alt="Script phase taking 80 percents of total time" title="Script phase taking 80 percents of total time" src="/static/66e61201f0f0d61019b40124f9789aa0/50383/performance-devtools.png" srcset="/static/66e61201f0f0d61019b40124f9789aa0/1d79a/performance-devtools.png 185w, /static/66e61201f0f0d61019b40124f9789aa0/1efb2/performance-devtools.png 370w, /static/66e61201f0f0d61019b40124f9789aa0/50383/performance-devtools.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </span> </div> <p><br> I then used the React profiler to see if this long scripting task comes from long react render times or useless component re-renders. The profiler is included in the React Dev Tools Chrome extension. <strong>It records renders, why your components rendered and for how long</strong>.</p> <p>Long story short, you mainly need to understand that a <strong>colored bar means that your component rendered, grey bar means it did not</strong>.</p> <div class="column-list"> <div style="width: 50%" class="column"> <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; "> <a class="gatsby-resp-image-link" href="/static/e92d74128a5a695b6e3c40302328ab3a/50383/profiler-render.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 28.10810810810811%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAGCAYAAADDl76dAAAACXBIWXMAAAsTAAALEwEAmpwYAAABQ0lEQVQY033Qu07CcBQG8L6JFxQsFQWJDgYlDKJvoKBGicFL4qANcZAiF4MOBjC6qc8igyR0MCExKokILYRESltI2qHj57/dTMDhN5zlO985VDgZQ4hjcZTL4PAmhb2rOI7vr8E+FXD6mLewxMlDDvu3l4gWhshncHCXBTVmd2FjK4LXSgVv7x/oKgokWcGP1CUki9LrofpVgy+wCjvjAeNegHN2fiBqZJxGeDMCoSGgVvtGU2xCFEX0+33oug5VVaFpGgRBhG95BbZJF2hmDg6nZyBq1MZgPbQDnufxUirhuVhEucxbPqtVEqbDMAwIZJEZOOGYAT3txRQJHcRquL0bhSzLpEUDrVYLnU4H9Xod7XYbEjnZnAXS2h9Yg4286N9Ac+OSP4hkOgsukQF38dc5l0I8kUbsLA63dxF22j00zPQLLAEg8J+3U1gAAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"></span> <img class="gatsby-resp-image-image" alt="profiler when component renders" title="profiler when component renders" src="/static/e92d74128a5a695b6e3c40302328ab3a/50383/profiler-render.png" srcset="/static/e92d74128a5a695b6e3c40302328ab3a/1d79a/profiler-render.png 185w, /static/e92d74128a5a695b6e3c40302328ab3a/1efb2/profiler-render.png 370w, /static/e92d74128a5a695b6e3c40302328ab3a/50383/profiler-render.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </span> <span style="font-size:14px;color: #b3b3b3">It did render</span> </div> <div style="width: 50%" class="column"> <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; "> <a class="gatsby-resp-image-link" href="/static/98d0b5384c4693980e2540e90c8ef450/50383/profiler-no-render.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 17.837837837837835%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAECAYAAACOXx+WAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAxElEQVQY02PwD4r8b23n/t/GwQOIgbQ9FAP51kDa3tn7v6tn4H9HFx8wG4TBau3dMTFQnEHP0OK/lq4JBtbWM/2voq73v3/ilP9r1q7/f+jQkf/nL1z6f/r02f/2Th7/VTUMwGrQ9TGoaxn+19A2+g+ikbGmjvF/eSXN/1u37fj/6uXL/2/evPn/7t3b/1+/fv3v4RXwX0FZG6wGXR8DyDBsGKRYRV3/f2Jyxv/a+ub/FVV1/0vLq8G0ibkd0IX6YDXo+gDSSJaQbUYaDwAAAABJRU5ErkJggg==&apos;); background-size: cover; display: block;"></span> <img class="gatsby-resp-image-image" alt="profiler when component does not renders" title="profiler when component does not renders" src="/static/98d0b5384c4693980e2540e90c8ef450/50383/profiler-no-render.png" srcset="/static/98d0b5384c4693980e2540e90c8ef450/1d79a/profiler-no-render.png 185w, /static/98d0b5384c4693980e2540e90c8ef450/1efb2/profiler-no-render.png 370w, /static/98d0b5384c4693980e2540e90c8ef450/50383/profiler-no-render.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </span> <span style="font-size:14px;color: #b3b3b3">It did not render</span> </div> </div> <p><br> And guess what? <strong>When an item in the list is clicked on, all the items contained in the list re-rendered</strong> 🥵 ! When the list is only 3 components long, it is ok to recompute but with 600 components, each click becomes quite expensive.</p> <p>For example here is the result in the profiler when the second card in the sidebar is clicked on. Basically, everything re-rendered.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 696px; " > <a class="gatsby-resp-image-link" href="/static/2484e3f2501873faef42e7e6fe7a66be/82158/profiler-initial-render.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 37.83783783783784%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAYAAAD5nd/tAAAACXBIWXMAAAsSAAALEgHS3X78AAABlElEQVQoz4WQ204aYRRGeQ0tSU1RHECGcQ4dDfEEGBmLaI3plbZVQUXTC6PWxKTpyUNtSbCCWGZqqxJC60su/xkTYkwMFytrZ1983872bb/V2Xqj49plZ1Hn07rJl3WD4uc0539eU71Y4MyZx64v4zRXqTXy2MLO/wL2vzVqYv59U+D0MofPH9DxB7QWTwSdzzQ6ulT80gDax33MUhn96Ifn4UqFVO0c6/qKyfo1Lxp1Ms0GUzdNrL8X+MKyTjiqCfQWEfmOcKQfJZVm+MMeE7UyE06VZCFHcm6alyf7zJweMls54pVd9Jg7O8Yn9en0RjRc38fbyQbdTyXMpRyjB18ZEchWlkBAZujbIcmfRRKlYxIn38V1v7CcyuOBHlGDYDCGmp3FzK1gLOeJjqQIxUzim5uoiXHG9t5jVUtk7DLpUrsLXYdVQuogsclpFBHslkix58jpKXp6ZIylPObGO+K7u6Jwpc2F98JDygCR+JhXIIk/S4rp7eWURd9QUhRmUTIz7QNblwo/nD1EgetgryLeMc4tVcUqCt3Wyf0AAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="initial profiler" title="initial profiler" src="/static/2484e3f2501873faef42e7e6fe7a66be/82158/profiler-initial-render.png" srcset="/static/2484e3f2501873faef42e7e6fe7a66be/1d79a/profiler-initial-render.png 185w, /static/2484e3f2501873faef42e7e6fe7a66be/1efb2/profiler-initial-render.png 370w, /static/2484e3f2501873faef42e7e6fe7a66be/82158/profiler-initial-render.png 696w" sizes="(max-width: 696px) 100vw, 696px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span> <span style="font-size:14px;color: #b3b3b3">Profiler view when an other card is clicked on, everything renders</span></p> <p>So the React rendering step needs to be lightened. What options do we got to perform this?</p> <ol> <li> <p>Find a way to <strong>reduce the number of items that need to re-render</strong></p> <p>It can be done thanks to virtualization. Virtualization is a technique to compute in the DOM only the elements located inside the user's viewport. When the user scrolls, elements will be added/deleted in the DOM. A few librairies already implement this for us <a href="https://github.com/bvaughn/react-window">https://github.com/bvaughn/react-window</a>.</p> </li> <li>Find a way to <strong>avoid useless re-renders</strong></li> </ol> <h2>Why virtualization is not the right solution here?</h2> <p>Virtualization is a good option when your rendering phase is taking a long time. But here, it’s the re-renders that take time. If re-renders issues are solved, displaying 600 items only once is not an issue.</p> <p>Moreover, the average user is making orders of about 50 items without a powerful computer. <strong>Virtualization is not very useful for short lists</strong>.</p> <p>Finally, the height of each item card is variable (when a field is filled, the item card will grow), and the number of item is variable (the list can be filtered, items can be created or deletes). <strong>Windowing librairies are not well suited for variable items size or number</strong>, some custom functions have to be added to make it work. For the user <strong>this can lead to a laggy scroll</strong>. And re-renders issues won’t be solved.</p> <p>So let's solve those useless re-renders!</p> <h2>Why do components re-render when another card is clicked on?</h2> <p>The structure is the following: <code>OrderItemsSelection</code> is the <strong>parent component</strong>, it contains the <code>Form</code> section and the <code>Sidebar</code>. The <code>Sidebar</code> itself has children: the <code>SidebarCards</code> (as many as items).</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/1c7f8747dc5febd83ebc9364b44c6dab/50383/tree-app-before-clicking.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 61.62162162162163%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAAAsTAAALEwEAmpwYAAABo0lEQVQoz5VSiW7bMAz1///TNgzohi07WhRFMzRZzjq+JcvyJdvBG8VYhpcWBSrggSJF8Xikl5cNriF1zbIxHdquR90aFGWNpnV6h5x8pH7513sr4Pl8hj2tMZB5jmEYWB+G8zsCWsdR5lSVortuDERRQddm9KunpLmTrwV0TnIM1pgeFQUL4gwHP0ScSabBtix1PWH+15NXGRxU1XB7u+MJi7sH/Lpf4vvtA4IoYbvzcRX/V6HLJIqLTFWJWBRIc41EXmSQSJYWGb0nUiOmt4T0ebVThTZb3fZougFhKpkby5sNGAkLNcl5siwvURItBf1X85btiZIMy/UW949PeNrsabIF220yVbUM69v1l2n3fQ8/CFnaY2ilPDcAQ5WdohSbY4DN4YT13keUSh7MvCULXbe8j6qseGCaElq9JPvE4XMsuCXL3zHKmCOr+2TPVPViC8R4L8ZVEo5DTW38WW2w2u55HQS1+Xd3gNIVZ30+RYRwWpXrNWPbzO751ObHLz/w6etPHsZyvcOHmwUeV1sIpXGzuMPnb78RJoI5nO/ea/gHxx6UYGYtduAAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="components full tree" title="components full tree" src="/static/1c7f8747dc5febd83ebc9364b44c6dab/50383/tree-app-before-clicking.png" srcset="/static/1c7f8747dc5febd83ebc9364b44c6dab/1d79a/tree-app-before-clicking.png 185w, /static/1c7f8747dc5febd83ebc9364b44c6dab/1efb2/tree-app-before-clicking.png 370w, /static/1c7f8747dc5febd83ebc9364b44c6dab/50383/tree-app-before-clicking.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span> <br> <code>OrderItemsSelection</code> has <strong>a state</strong>, <code>selectedItem</code>, and a <strong>state setter</strong>, <code>setSelectedItem</code> for the <strong>selected item id</strong> (thanks to a <code>useState</code> hook).</p> <p><code>OrderItemsSelection</code> passes <code>selectedItem</code> as a prop for the <code>Form</code> section and the <code>Sidebar</code>.</p> <p><code>OrderItemsSelection</code> also passes <code>setSelectedItem</code> setter to the <code>Sidebar</code>, <code>Sidebar</code> passes the <code>setSelectedItem</code> setter to its <code>SidebarCard</code> children to be used on card click.</p> <p>So when the <strong>red shirt card is clicked on</strong>, <strong>a new state is setted</strong> with <code>setSelectedItem</code>, the selected card is now the red shirt card.</p> <p>As the selected card is a state of the component <code>OrderItemsSelection</code>, it re-renders: a <strong>component state change triggers a re-render</strong>. Therefore <strong>all its children also re-render</strong>: the <code>Form</code> section, and the <code>Sidebar</code>. As the <code>Sidebar</code> <strong>re-renders, so are its children: the white/red/blue shirts cards</strong>.</p> <div class="column-list"> <div style="width: 60%; padding: 10px" class="column"> <div style="padding: 10px"> <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; "> <a class="gatsby-resp-image-link" href="/static/d403953b2876251bfeb2d1cc9e181bc3/50383/profiler-global-initial-state.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 37.83783783783784%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAYAAAD5nd/tAAAACXBIWXMAABYlAAAWJQFJUiTwAAACF0lEQVQoz1WQSU9TYRiF+0cwmogd7tRLaZ0qpeEi0ASQ4SIITilIIjiLMaRYCEW9FDowQ0uBLoCiRhMjcWNiNC50YUKMLkzc8A/0Dzx+FjRxcfK833lzTr68tsDkGBXWCIGJGFXT4xhzk1QvzxJcWSK4mhHaY2VxzuLPLdL1tED09TaDr14yJNiwnqdiLUsgM49Nn0uhpeNoCQvPbLIo/1IScytF65MpTKHWQor2zQSdhQRn1i0yb5/z8/s3dr985tfuDyLbq7RsxGnPW9hOx+7REu2jY36YqjkLV3aJith9rpg63R0+wmYZtwYbGVvrZzR3lUi2h40Xk+x8/MD7d2/4uvOJhUKE4Xw/0dw1bJpRi6p7ObGQxjPyAP1hDCncS8lBNyWl5ZQcUDl0qo7S6RnUZAI9/hhjcYqLhSznt5a59GwVY3MN/0Ye/0oGm9MIcUT2Inf3Il/uRgn3oJzrwqV4cKlenE43crAaT3KCo2dNgnf6MHJp/DNxgosJKjMpcfdxapdT1OSmRKHXj0PxIjU0Ize3IZsdSI0t2F0e4Zdjt7txBarRrt/gsOJDD9VTGY8hR4ZQbt5GjY2iWI9Qxy206TQ2hyxCUjnOk8FiUKqtR6pvEoVl7O1Eseor7v7QIXyXUYfcZCKFGlHEB9wDd1E6L6ANDPwtFCH3MRz6cRyarxj45+/T7tT/f+/LIQsKT6uqoay1jd/T91Pt1cCKVAAAAABJRU5ErkJggg==&apos;); background-size: cover; display: block;"></span> <img class="gatsby-resp-image-image" alt="profiler on app initial state" title="profiler on app initial state" src="/static/d403953b2876251bfeb2d1cc9e181bc3/50383/profiler-global-initial-state.png" srcset="/static/d403953b2876251bfeb2d1cc9e181bc3/1d79a/profiler-global-initial-state.png 185w, /static/d403953b2876251bfeb2d1cc9e181bc3/1efb2/profiler-global-initial-state.png 370w, /static/d403953b2876251bfeb2d1cc9e181bc3/50383/profiler-global-initial-state.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </span> </div> <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; "> <a class="gatsby-resp-image-link" href="/static/631be6f260417467e056f6af4b4df5a9/50383/components-hooks.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 55.13513513513514%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAABYlAAAWJQFJUiTwAAABPUlEQVQoz5WT2ZKDIBRE/ZapjMuERUFQccmmZh4m//85PRcSU0lVlsrDqUYKmuZejKSqkBOFroNyaaBzG8a5n7MdpOnAaY6JMsBliTSRiOM1vhMGIQyU6cOeKC87BMxZhXZk3tK4RUGLZDmAqRascEQT4ISgNWtZI2UGCSuRcRO+I10NeEZBhxh7gOsmdNsNmn4P0+xIR9h2D6EckrUm0zKoJ1K2xyP0RX26xKfgCqtM4Ss9E//oq5knI7iwzxN6M1sPOI0z/sYNTlN7ZvZKyam+8Y3hwhNDSueVGjLuJkyHGb3r4aom0FYOQtq7hC8Nw5UvNZTDL1KzxYqu6K+8sKRLPjHMqeO8P4I1h8uG+0S3CZfx64T0dHg3IyvctYvveJ2QDEu3R7eZ6WHXD2v2UQ1Dp5tt0HdG/rCM/ph/f6ZgKgdR0e0AAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"></span> <img class="gatsby-resp-image-image" alt="components hooks" title="components hooks" src="/static/631be6f260417467e056f6af4b4df5a9/50383/components-hooks.png" srcset="/static/631be6f260417467e056f6af4b4df5a9/1d79a/components-hooks.png 185w, /static/631be6f260417467e056f6af4b4df5a9/1efb2/components-hooks.png 370w, /static/631be6f260417467e056f6af4b4df5a9/50383/components-hooks.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </span> <span style="font-size:14px;color: #b3b3b3">In &quot;components&quot; tab the value attributed to Hook 1 is displayed (which caused a render)</span> </div> <div style="width: 40%; display: flex" class="column"> <div style="align-self: center"> <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; "> <a class="gatsby-resp-image-link" href="/static/21b8f2333c133f31a60cc2722661b09f/50383/tree-app-after-clicking.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 61.62162162162163%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAAAsTAAALEwEAmpwYAAACSElEQVQoz5WSS2tTURSFM9BfINSBOHFqB/6C4kBQHOvfcCAIHSiIgrQDB1pQkSBVKg58ICiCrRWc1NY09pG0JaVt2pukvffmcc995ubVz50Tg6U+wMFisw97rbP2I6GCOofh+JGO9UaTuNkiiht4QUQ97udNlNQ4/u/cxL8EO+0m0CaOhexUabdinbdajZ5g8F+CMUr4bgP8FtTqvdjNu+8HP/6roC7QPzdw3Cp+bgQ3M0wpdZ38zDBm+gZe9hrexpjUBtphX7QfE9q2FgoFUuT7qFBackrEs2covjjF2pOz5J4NsZIcwnx5knr6HMrzcDU30OJKIxSH2lGME4ET9lD197FVk7K9hb2Xo7ybZdf4LjGDXVqQ903KXpuy26bid1Bdrua3+w49VDmLX1sSLFIyZqmY36hYImCuYwnDUg1BjOW2sOyCCM9jldKUJXqVLMpeQtV2RFBUncom8ZcB7NcnyE8Mknt6GuP5ILW3x4hnBlFqD1fG4HY/luUEK1dpTh8l/Hyc/JsBgukB4qkj+JkrXYfSrmPir9/BTN2lOJfEmH2I8fUxVnpEFnBPL8jRdxfqtlThA97aTWqrIxQXRnHWRvElV4X3MsOoQ036Lzpg+lCWYy1UPGktwvSgpGSmgZxJtP/rCsKOnlnXrRf3Ym+GHRLexiO2Pl5ge+o84dYD7OIimfl3ONYq4XaS/KeLGvpUuqSf2+xv1zm8ZWthjKXxSyyPX6a6fJ+d1CTzr5Jsz00SGBMYqVsU07dxN5PiYL/X9h8Ouo8fJAFo9ooKzowAAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"></span> <img class="gatsby-resp-image-image" alt="app initial state updating schema" title="app initial state updating schema" src="/static/21b8f2333c133f31a60cc2722661b09f/50383/tree-app-after-clicking.png" srcset="/static/21b8f2333c133f31a60cc2722661b09f/1d79a/tree-app-after-clicking.png 185w, /static/21b8f2333c133f31a60cc2722661b09f/1efb2/tree-app-after-clicking.png 370w, /static/21b8f2333c133f31a60cc2722661b09f/50383/tree-app-after-clicking.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </span> <span style="font-size:14px;color: #b3b3b3">Component updating schema</span> </div> </div> </div> <p><br> But in fact what do we truly need to render in the sidebar? If we take a closer look, the blue shirt card <strong>does not need to be rendered as none of its props actually change</strong>.</p> <h2>How to prevent re-renders</h2> <h3>What is memoization?</h3> <p>Let’s take a look at Wikipedia definition:</p> <blockquote> <p>In computing, <strong>memoization</strong> or <strong>memoisation</strong> is an optimization technique used primarily to speed up computer programs by storing the results of expensive function calls and returning the cached result when the same inputs occur again.</p> </blockquote> <p>So here, the <code>Sidebar</code> is a parent component of each card component. So we want to <strong>cache the blue card component computations</strong> because it has unchanged prop values, ie <strong>memoize this component</strong>. We can find more information about component memoization in the React documentation:</p> <blockquote> <p><a href="https://reactjs.org/docs/react-api.html#reactmemo">React documentation</a>:</p> <p><code>React.memo</code> is a <a href="https://reactjs.org/docs/higher-order-components.html">higher order component</a>. If your component renders the same result given the same props, you can wrap it in a call to <code>React.memo</code> for a performance boost in some cases by memoizing the result. This means that React will skip rendering the component, and reuse the last rendered result.</p> </blockquote> <h3>Memoize a React component with React.memo()</h3> <p>Ok so if everything goes as planned <strong>I just have to add the memoization to <code>SidebarCard</code> component</strong>, and React will magically understand that if props values didn’t change, the component does not need to re-render.</p> <div style="display:flex;flex-direction:row;align-items:center;justify-content: space-around;"> <div> <div class="gatsby-highlight" data-language="jsx" style="margin-right: 10px;"><pre class="language-jsx" style="padding: 20px; font-size: 9pt" > <code class="language-jsx"><span class="token keyword">export</span> <span class="token keyword">default</span> SidebarCard<span class="token punctuation">;</span></code></pre> </div> <span style="font-size:14px;color: #b3b3b3">Before memoized component</span> </div> <div> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx" style="padding: 20px; font-size: 9pt"> <code class="language-jsx"><span class="token keyword">export</span> <span class="token keyword">default</span> <span class="token function">memo</span><span class="token punctuation">(</span>SidebarCard<span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre> </div> <span style="font-size:14px;color: #b3b3b3">After memoized component</span> </div> </div> <p><br> But it would be too easy won’t it be?</p> <div class="column-list"> <div style="width: 50%" class="column"> <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; "> <a class="gatsby-resp-image-link" href="/static/cc31fe1b7d5103efae830065f5e8f840/50383/tree-sidebar-expectations.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 65.4054054054054%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsTAAALEwEAmpwYAAADAklEQVQ4y42S22scZRjG91oo9K5X8VgJFLyoFilRUfSmYCzYGlFQhCJi/QeE2gvF2laQHrRJD5rYbD3iKawFg80mZrfJ2uawKW6aZLuHzMweptnZ/WZ2Z2dit/DznSFavBC8ePje553v/fG8wxexXZ9QLQ/l/ontNHC0b2gWo7T0KK4xTD0/TEPk6oGiNFeH5c632E0lM+vh7N+cyB1gG9W+jd3Q8JLd+BNbsEfvwfnlXoo/dqOPdOOM3ocS741vkTvbsFVJZjrh7H8CVUPHm96Bl3iQmY928du7L5I+/TrpM2+QOPQS04d7aca78X5/FKXK/xM4tR1v8n5SR3bx69t9JI+8xuUP9zF2oI/EewIc24qfeuRfQLWxdiQoQhMCOyGwPfUw7ckHyF3YSebsY2SHnyIbfZrFoD7fQ2t8K23ZQjUE6N4BBoqEcVuuGBfHFXDTxjaviFK0rCSeSmJVxqiLPHUZV3rBN2XOykxLUq3/AwuBYSoPmutg1Hz0NR+jIbXVQbegpCBb7nCjcpuS9PWa9CyfSt2nvKao1Bq4/i2a3roECv5hbRGnMEQ99wVGZpBaNoqTP4da+wPbuIjKfUbl+hDVpSGpB3H0nzCtOjnDZHm1zLWVVbJaVfxNzLpDxCl8TiexCe3nLpa/76I62kVn8i5U/jz+/G7a45tY+eFu8iNdeBObuTX3DHmjRHxmiZF4iq8vThCbuMrYlQw3dFMSFr5k6cwO5vt7WRh8gflTveK309QuUL4kz+V4Dwuf7gmVPvEE5qWXWa0Yks5kpVhiej7DcqHEykbKEDj9/pNMybNIn32L5NF9pA49TsuIko+9SfzAs8wO7Ge2fz/xd3ajxV5Fq+ropkIrmyxcz1I0qhhmnXwpAMpqVz94iNTR50gP9IUPd/bwNlraIMXYKyQP7mTuk73MfbyXxMEejNjzklCTNDXyepmZa0vktEoIC1dWN9PUF49jLfZjZU7JOSD+GEFfFb+T3glR/4ZOYhe+omo1ZEUzhORLa+EZrFyp2fwF8Vtpzyl9g08AAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"></span> <img class="gatsby-resp-image-image" alt="expected not rendered third component schema" title="expected not rendered third component schema" src="/static/cc31fe1b7d5103efae830065f5e8f840/50383/tree-sidebar-expectations.png" srcset="/static/cc31fe1b7d5103efae830065f5e8f840/1d79a/tree-sidebar-expectations.png 185w, /static/cc31fe1b7d5103efae830065f5e8f840/1efb2/tree-sidebar-expectations.png 370w, /static/cc31fe1b7d5103efae830065f5e8f840/50383/tree-sidebar-expectations.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </span> <span style="font-size:14px;color: #b3b3b3">Expectations</span> </div> <div style="width: 50%" class="column"> <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; "> <a class="gatsby-resp-image-link" href="/static/7e5720801abe533be7292c3d2b9e1ce6/50383/tree-sidebar-reality.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 65.4054054054054%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsTAAALEwEAmpwYAAADA0lEQVQ4y42SXWibZRTHc+MuhF6XXdTPucLAi+mQsQnKbiysiqxWvBCFoWK9FVSCiGXfIK5sbffV1iZtYcMpoxUsLuts0zZzdknHkkzbpGm+Oteked6vvOnIxs/zPp0MLwQv/rznnOf8f885vI/PcGpo2S7KuYdhVjCz57EyQexcECcfYC0doCJycp6CWMsB6bmAYSnxrGvvPxzfI2AVVb2PUcnihpupXW3EGH8S8+enyPzYTO5SM+b40yjJ3YlG6dmGoQriqWvvfwJVJYc7uwN36jl+/6aFX79+m9ipD4id/oipg+8we7gV60oz7rWXUKr4P4Ez23EnnyFypIVfPm8nfOR9po/tJ+RvZ6pTgKEt1CIv/guoHq7t8wKdaGBdA6szL1CdfJbU0E7iZ3azEHiVheAeEl48uAt7YgtV2UJVBOg8Anry6XFtRxIH0xGwZWD89Zsogl0O46ow5ZUQayJXTeNIzZQz3WOZ2uN5Da2qTOhN5YK1DvlSjdxqjXxF4nKdXBkKChaKdRZX7lOQeq60caZ7SuvaY4rXY3gsn1FKYC4NsJYaJh/vp7QQxEyfRa3ewsj/hEr1sZIc4M7tAR3r2mpcPH2UFocpJMSbHsEShlGK4zOXvqM+1UB2rIk/LjZxZ7yJ+uTjqPQgtegbVCca+POHJ0hfasK92oAbfRNjaZh7oU3ylBpJXthMcbSR2uXHZJA+mXBphNundxDtaWW+/y2i3a2Sb8fKDlG8LM/l+C7mz+3bUNduiqEPsXIjPFj8FDvdzfL1TqzUCR6kPpOLhjaAswdeYUaeRezMJ4SP7idy8GXsfJD06Mdc8e9lrreDuZ4OJvytpMc6sAsjxM+1MT/oJ37+C24G/dw62y5bBQUoq10/9DyRo68T623XD3fu8DbsbD+Z0XcJf7mTGyfbuHGiTcfLY+/JZQGudW4l3Pka0ZP7mD7QQuSrrahFWVndjbGWOE450UM53i3fXsm/xaurzPdS6xL1PFQXlcxF+Sk3qSTFk+yVfvEkT+lc3Y3yN+SgV83QDiffAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"></span> <img class="gatsby-resp-image-image" alt="rendered third component schema" title="rendered third component schema" src="/static/7e5720801abe533be7292c3d2b9e1ce6/50383/tree-sidebar-reality.png" srcset="/static/7e5720801abe533be7292c3d2b9e1ce6/1d79a/tree-sidebar-reality.png 185w, /static/7e5720801abe533be7292c3d2b9e1ce6/1efb2/tree-sidebar-reality.png 370w, /static/7e5720801abe533be7292c3d2b9e1ce6/50383/tree-sidebar-reality.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </span> <span style="font-size:14px;color: #b3b3b3">Reality</span> </div> </div> <p><br> When we launch the profiler and focus on the blue shirt card:</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/93125956f82fae04d95a75fa9db6e273/50383/profiler-only-memo.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 30.270270270270274%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAGCAYAAADDl76dAAAACXBIWXMAABYlAAAWJQFJUiTwAAABnklEQVQY01WP30tTcRjGzx+yrKGcc/Y9Z+67M6fB3GFzORzZlIpqFSKETkRNxyRR8UKhH6jbtFpNx8Du6iIkNCL8L4QITJTI6EIUL7wTPp1NhHrhAw+8L8/zvErkyQixp5PEcrN0Fp/TkZ8jnnN4u0CinOdWucA9h+RaibbKKrHKCtfeVWhfeYX9ch57dRm7tIRdzBEpFVC8QhK4kSRcyhN9PIgUPqxkktbX84QHB9jc2OD7t21+Hvziz9ERvw8POT0748PHdcypCeLTGRKz47Quv8BaWkBRRQDdbCKcHUV23aTebWLIFpomxgneTrH1+Su7e/vs7Pzg2DE8OT6hOp82vyBmpvFaV5E9vfjKbxCFRZSqmar7MYMhDAfdG0R1C/zZLDL1kAd3exgYGqOvf4j0cIbM+nuyaxW6u+4QvJ/CIyxE7yNCxUXiz6ZQNOO8oeYsqlwEBBLXaQxFcNVpuNweXJd1LtUbBLo7MaNR6q540J37Wng6TXO6H58hzxtemP6rVdWH5vEjGpv/Q1Ulum7VdO0bJ1zabbXwBk3yF3Ek+mvUdzq3AAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Profiler with memoized component" title="Profiler with memoized component" src="/static/93125956f82fae04d95a75fa9db6e273/50383/profiler-only-memo.png" srcset="/static/93125956f82fae04d95a75fa9db6e273/1d79a/profiler-only-memo.png 185w, /static/93125956f82fae04d95a75fa9db6e273/1efb2/profiler-only-memo.png 370w, /static/93125956f82fae04d95a75fa9db6e273/50383/profiler-only-memo.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>We can see that <strong>components are memoized thanks to (Memo)</strong> just after component name, but we can also see that <strong>React considered that props did change for the last card</strong>: <code>sidebarItems</code> and <code>onSetSelected</code>. What’s interesting is that it recognized that the <strong>isSelected boolean did not change</strong>. It’s a boolean so the comparison is only made based on the boolean value. <strong>With functions, array and object</strong>, it’s different. Even if the objects have the same properties, they are <strong>considered equal only if it refers to the exact same object</strong>.</p> <div class="gatsby-highlight" data-language="tsx"><pre class="language-tsx"><code class="language-tsx"><span class="token keyword">const</span> Laura1 <span class="token operator">=</span> <span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token string">"Laura"</span> <span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token keyword">const</span> Laura2 <span class="token operator">=</span> <span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token string">"Laura"</span> <span class="token punctuation">}</span><span class="token punctuation">;</span> Laura1 <span class="token operator">===</span> Laura2<span class="token punctuation">;</span> <span class="token comment">// => false</span></code></pre></div> <p><strong>Functions, as well as arrays and objects are stored by reference</strong> into memory and not by value. We call these <strong>non-primitive data types</strong> as <strong>opposed to primitive types</strong> (string, boolean, number, undefined or null) which are stored by value. Here is <a href="https://javascript.plainenglish.io/javascript-reviewing-primitive-and-non-primitive-data-types-5bc4ca68c3de">a great article</a> to understand all about it.</p> <h3>Memoize a function with useCallback()</h3> <p>This means that a new function <code>onSetSelected</code> was created earlier, which has the <strong>same value</strong> as the former <strong>but not the same reference</strong>. They are <strong>two different entities</strong> even they look alike.</p> <p><code>onSetSelected</code> is a function that is <strong>called when a card is clicked on</strong> to set the <strong>new selected item id</strong>.</p> <p><code>onSetSelected</code> is passed from the sidebar to the card here:</p> <div class="gatsby-highlight" data-language="tsx"><pre class="language-tsx"><code class="language-tsx"><span class="token keyword">const</span> Sidebar <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">{</span> onSetSelected<span class="token punctuation">,</span> <span class="token comment">// => received here</span> sidebarItems<span class="token punctuation">,</span> selectedItemId<span class="token punctuation">,</span> allowMultiSelect <span class="token operator">=</span> <span class="token boolean">false</span><span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token operator">:</span> SidebarProps<span class="token punctuation">)</span><span class="token operator">:</span> <span class="token constant">JSX</span><span class="token punctuation">.</span><span class="token parameter">Element</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token punctuation">(</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>div</span><span class="token punctuation">></span></span><span class="token plain-text"> </span><span class="token punctuation">{</span>sidebarItems<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter">sidebarItem<span class="token punctuation">,</span> index</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">(</span> <span class="token operator">&lt;</span>SidebarCard sidebarItem<span class="token operator">=</span><span class="token punctuation">{</span>sidebarItem<span class="token punctuation">}</span> positionIndex<span class="token operator">=</span><span class="token punctuation">{</span>index<span class="token punctuation">}</span> onSetSelected<span class="token operator">=</span><span class="token punctuation">{</span>onSetSelected<span class="token punctuation">}</span> <span class="token comment">// => passed here</span> isSelected<span class="token operator">=</span><span class="token punctuation">{</span>selectedItemsId <span class="token operator">===</span> sidebarItem<span class="token punctuation">.</span>id<span class="token punctuation">}</span> <span class="token operator">/</span><span class="token operator">></span> <span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">}</span><span class="token plain-text"> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>div</span><span class="token punctuation">></span></span></code></pre></div> <p><br> <code>onSetSelected</code> is defined in the <code>Sidebar</code> parent:</p> <div class="gatsby-highlight" data-language="tsx"><pre class="language-tsx"><code class="language-tsx"><span class="token keyword">const</span> OrderItemsSelection<span class="token operator">:</span> React<span class="token punctuation">.</span><span class="token function-variable function">FunctionComponent</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> <span class="token punctuation">{</span> initialValues<span class="token punctuation">,</span> submit<span class="token punctuation">,</span> selectedItemId<span class="token punctuation">,</span> setSelectedItemId<span class="token punctuation">,</span> <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">useItemsSelection</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// => onSetSelected definition</span> <span class="token keyword">const</span> <span class="token function-variable function">onSetSelected</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">index<span class="token operator">:</span> number</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token function">setSelectedItemId</span><span class="token punctuation">(</span>index<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token punctuation">(</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">Formik</span></span> <span class="token attr-name">initialValues</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>initialValues<span class="token punctuation">}</span></span> <span class="token attr-name">onSubmit</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>submit<span class="token punctuation">}</span></span><span class="token punctuation">></span></span><span class="token plain-text"> </span><span class="token punctuation">{</span><span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> values<span class="token punctuation">,</span> validateForm <span class="token punctuation">}</span><span class="token operator">:</span> FormikProps<span class="token operator">&lt;</span>ItemsSelectionValues<span class="token operator">></span></span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token punctuation">(</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span></span><span class="token punctuation">></span></span><span class="token plain-text"> &lt;Sidebar sidebarItems=</span><span class="token punctuation">{</span>values<span class="token punctuation">}</span><span class="token plain-text"> onSetSelected=</span><span class="token punctuation">{</span>onSetSelected<span class="token punctuation">}</span><span class="token plain-text"> // => passed here selectedItemId=</span><span class="token punctuation">{</span>selectedItemId<span class="token punctuation">}</span><span class="token plain-text"> /> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">ItemsSelectionForm</span></span> <span class="token attr-name">initialValues</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>values<span class="token punctuation">.</span>items<span class="token punctuation">[</span>selectedItemId<span class="token punctuation">]</span><span class="token punctuation">}</span></span> <span class="token attr-name">selectedItemId</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>selectedItemId<span class="token punctuation">}</span></span> <span class="token punctuation">/></span></span><span class="token plain-text"> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span></span><span class="token punctuation">></span></span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token plain-text"> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span><span class="token class-name">Formik</span></span><span class="token punctuation">></span></span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre></div> </sub> <p>So when the <code>OrderItemsSelection</code> component re-renders, a new <code>onSetSelected</code> is generated. The blue shirt card receives a new function (weirdly same looking as the previous one). So it has to re-render.</p> <p>We want to avoid recomputing the function, as nothing has changed except the re-render in <code>OrderItemsSelection</code>. In short, we <strong>need to memoize the function</strong>. Fortunately, React has already implemented this for us. A big welcome to the beautiful <code>useCallback</code> <strong>hook which will memoize our function</strong>!</p> <div style="display:flex;flex-direction:row;align-items:center;"> <div> <div class="gatsby-highlight" data-language="tsx" style="margin-right: 10px;"><pre class="language-tsx" style="padding: 20px; font-size: 9pt"><code class="language-tsx"><span class="token keyword">const</span> <span class="token function-variable function">onSetSelected</span> <span class="token operator">=</span> <span class="token punctuation">(</span>index<span class="token operator">:</span> <span class="token builtin">number</span><span class="token punctuation">)</span> <span class="token operator">=&gt;</span> <span class="token punctuation">{</span> <span class="token function">setSelectedItemId</span><span class="token punctuation">(</span>index<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre> </div> <span style="font-size:14px;color: #b3b3b3">Before without useCallback</span> </div> <div> <div class="gatsby-highlight" data-language="tsx"><pre class="language-tsx" style="padding: 15px; font-size: 9pt"> <code class="language-tsx"><span class="token keyword">const</span> onSetSelected <span class="token operator">=</span> <span class="token function">useCallback</span><span class="token punctuation">(</span><span class="token punctuation">(</span>index<span class="token operator">:</span> <span class="token builtin">number</span><span class="token punctuation">)</span> <span class="token operator">=&gt;</span> <span class="token punctuation"> {</span> <span class="token function">setSelectedItemId</span><span class="token punctuation">(</span>index<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre> </div> <span style="font-size:14px;color: #b3b3b3">With useCallback</span> </div> </div> </div> <p><br> Now it’s time to try if our <code>onSetSelected</code> is still a guilty prop.</p> <div class="column-list"> <div style="width: 50%" class="column"> <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; "> <a class="gatsby-resp-image-link" href="/static/8512189f28323d189d27b508bd3389f0/50383/profiler-only-item-props-changed.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 42.7027027027027%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAYAAAAywQxIAAAACXBIWXMAABYlAAAWJQFJUiTwAAACIElEQVQoz5WQyU8TYRiH5x/xUsPMdJlpO1NAijTIvowslhYiViMgSAuxFYZuLGI0VA1YxIIJIZqQEBO9GqMnjQkHz+LR9aSeNPH8+LUJCRz9kie/N+/35vcuUq9l0d/by2AkSnQgUmFIxEORyP8RjRIJDyDFLjQTi7Vy8VIbo6OdjI10Er9qEZ88e5wjucRkD9Px40wJJsa7kbypFPrUFN5EAn8yiW/WxshkMbJ5oTn8AiObI7CwSCCXJyBi71waVdTJMzZV12craOkMhp1Gark2TksuSeuDW3Tv3Kd6tYBeKuLbWsfYLmEI9T/aILiQobFYILi1xtjzXVZev2Dp1UsK795yVxB+toe2sYqkOlx4+sKYOyXM28u4rD7cy4to66ucSk3TsLJE7c4mrvozeDNp5M0id57u8evLV759+szf338ov8LuE/T5WSTZaaCHmgkmJqiJTyI7PHjOD2OW1nA3NOPJZQmmZ3DrAbShYZSHRdKFe3w4OODN/j7vPx7w/cdPknN5HKmkmNATQHH6UVwmrqYOoQZaXYjTdgqtw8JYzOFu7aLqhCI2GUC3bWrCg3SIuzXFLtP+eJs2cV/T6sc9ZyMpbpOy6aFxRRUfSk0IZ22I6i4LT20Diir+zCD6yBXUdouTgXocot65NI/ccw7ZqEPLZ4Vh2eyo6WEsJi1PK5fNhapaNYrsrTRxCjNV5J2iznfzBu7GNlTRUJ/P8w8TrXK2+D6PNQAAAABJRU5ErkJggg==&apos;); background-size: cover; display: block;"></span> <img class="gatsby-resp-image-image" alt="Profiler only item props changed" title="Profiler only item props changed" src="/static/8512189f28323d189d27b508bd3389f0/50383/profiler-only-item-props-changed.png" srcset="/static/8512189f28323d189d27b508bd3389f0/1d79a/profiler-only-item-props-changed.png 185w, /static/8512189f28323d189d27b508bd3389f0/1efb2/profiler-only-item-props-changed.png 370w, /static/8512189f28323d189d27b508bd3389f0/50383/profiler-only-item-props-changed.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </span> <span style="font-size:14px;color: #b3b3b3; align-self: center">Profiler with memoized onSetSelected function</span> </div> <div style="width: 50%" class="column"> <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; "> <a class="gatsby-resp-image-link" href="/static/a44addeb50f88823117849065a76becb/50383/tree-sidebar-usecallback.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 65.4054054054054%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsTAAALEwEAmpwYAAAC+0lEQVQ4y42TT2wUVRzH96IXI0cFEnugMUQORopIYoh35ICe9ELU4sGjiTHevJhoTNF2iyDgbtttl2oxGKtoRKhgcdug22633YH+24XZ2VZ2Z3fmzezObGWbfPztK4Z4MR6++f15v+/3fd+bNxHVaKJRD3Ebf6E8B8/8Cv/2MA1rhKA0TK2QwBEE1lbPv5OQmTGU7wpnQ3P/0Yk8EAxwg02UYxJe303z6uPUvttB7dsdFM53Yl7oxBnfSVXq8JfHCH/bg+uWhNPS3P8QLBJOPUs4+SRLo93MDb2LMfYeOUE2IfnIW9Sv7SGcfk4E1/6nYOoZEdxFZepDrOuDGD9+Su6HTzB//YLydJRg8inZtOtfgu79Y0faiS60YAtXBINUF8G1XSzG9jLXt4/l+EFWBl8gG92HcbqL+kQnwdR+mRXBxgPBNiLabr0hRQOvIcK+Qt29IZim4aRoeimq61eorU/QVCkC6Xmypmd8T3PaXKURiMO2qxD8DbDsJsVKE8uRvNqiWIWSC8trLVbWN3VetLfW9Iy9oTmecNsaba2Isg28wgC11SRWLo69PIyXP4tbWUBZF3FXY6zfHODPWwM6171KTjgx7JUkJUO4+XP4oqHsHBGvMMjm5KOYFztYvPAEd3/qkPoR3PwQzcwRwqvbWPqmg/x4hzylbYSzL6EKSe5NPExxfDu3xnay9v12mpcfEiMxcVg4h/HZXtLRQ2TOvqyjcfJpfHOE0s/dzPQcYPb0ETKC2eMHKF0+hl8cpbX4Nt5yH4Xp91FLvbSW3pGNRsThnS9x0h9h3hhmNXWK4u9JnD8+QOXjqIWPKafj5KfOaJTTMd1TcqqWJddiXaFojFIzL3HPkvnbSXEoR8udPMjcmaPcTHaT+fwoxqnn5feKUbp0jNm+w+QSb7Aw9DozvS+K6zepm0PMR/eT7n8FI/EaMydeZa5XntFqXL5yOYOdPU4l2y+I6mhne+TiMzj581QyPffX+nVey3+NqmSpzgtn/sQWR2K7dssz/A3muWcALoXuegAAAABJRU5ErkJggg==&apos;); background-size: cover; display: block;"></span> <img class="gatsby-resp-image-image" alt="Schema only item props changed" title="Schema only item props changed" src="/static/a44addeb50f88823117849065a76becb/50383/tree-sidebar-usecallback.png" srcset="/static/a44addeb50f88823117849065a76becb/1d79a/tree-sidebar-usecallback.png 185w, /static/a44addeb50f88823117849065a76becb/1efb2/tree-sidebar-usecallback.png 370w, /static/a44addeb50f88823117849065a76becb/50383/tree-sidebar-usecallback.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </span> <span style="font-size:14px;color: #b3b3b3; align-self: center">Component updating schema</span> </div> </div> <p>Nice shot, <code>onSetSelected</code> is not guilty anymore, but we still have one targeted prop: the <code>sidebarItem</code> .</p> <h3>Use React.memo() comparison function to tell React when a component should re-render</h3> <p>Same protocol, as done before, where is <code>sidebarItem</code> defined?</p> <div class="gatsby-highlight" data-language="tsx"><pre class="language-tsx"><code class="language-tsx"><span class="token keyword">const</span> Sidebar <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">{</span> onSetSelected<span class="token punctuation">,</span> sidebarItems<span class="token punctuation">,</span> selectedItemId<span class="token punctuation">,</span> allowMultiSelect <span class="token operator">=</span> <span class="token boolean">false</span><span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token operator">:</span> SidebarProps<span class="token punctuation">)</span><span class="token operator">:</span> <span class="token constant">JSX</span><span class="token punctuation">.</span><span class="token parameter">Element</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token punctuation">(</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>div</span><span class="token punctuation">></span></span><span class="token plain-text"> </span><span class="token punctuation">{</span>sidebarItems<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter">sidebarItem<span class="token punctuation">,</span> index</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">(</span> <span class="token comment">// => defined here</span> <span class="token operator">&lt;</span>SidebarCard sidebarItem<span class="token operator">=</span><span class="token punctuation">{</span>sidebarItem<span class="token punctuation">}</span> <span class="token comment">// => passed here</span> positionIndex<span class="token operator">=</span><span class="token punctuation">{</span>index<span class="token punctuation">}</span> onSetSelected<span class="token operator">=</span><span class="token punctuation">{</span>onSetSelected<span class="token punctuation">}</span> isSelected<span class="token operator">=</span><span class="token punctuation">{</span>selectedItemId <span class="token operator">===</span> sidebarItem<span class="token punctuation">.</span>id<span class="token punctuation">}</span> <span class="token operator">/</span><span class="token operator">></span> <span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">}</span><span class="token plain-text"> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>div</span><span class="token punctuation">></span></span></code></pre></div> <p>When <code>Sidebar</code> renders it recomputes the map, and <strong>each</strong> <code>sidebarItem</code> <strong>is recomputed</strong>. Since it’s a brand <strong>new object</strong> and as explained earlier, <strong>with a new reference</strong>, so it is not equal to the previous object even if it contains the exact same values. So <strong>how can we tell our card component to take into account only deep equality of previous and current</strong> <code>sidebarItem</code>? Once again, React team has already a solution right in the <a href="https://reactjs.org/docs/react-api.html">React.memo doc</a>:</p> <blockquote> <p>By default it will only shallowly compare complex objects in the props object. If you want <strong>control over the comparison</strong>, you can also provide a custom comparison function as the second argument.</p> </blockquote> <p>Nice, the next step is to define a function that can compare the <code>SidebarCard</code> props in a customized way.</p> <h4>Stringify is not suitable for deep comparison</h4> <p>We could have thought to compare the stringifying version of the two objects, but it’s <strong>not the best idea</strong> as when we have <strong>equal properties not in the same order</strong>, it would have <strong>returned false</strong>.</p> <div style="display:flex;flex-direction:row;align-items:center;"> <div class="gatsby-highlight" data-language="tsx"> <pre class="language-tsx" style="margin-right: 10px;font-size: 6pt;padding-bottom:20px"><code class="language-tsx" style="font-size: 9pt;"> <span class="token keyword">const</span> Laura1 <span class="token operator">=</span> <span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token string">"Laura"</span> <span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token keyword">const</span> Laura2 <span class="token operator">=</span> <span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token string">"Laura"</span> <span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token constant">JSON</span><span class="token punctuation">.</span><span class="token function">stringify</span><span class="token punctuation">(</span>Laura1<span class="token punctuation">)</span> <span class="token operator">===</span> <span class="token constant">JSON</span><span class="token punctuation">.</span><span class="token function">stringify</span><span class="token punctuation">(</span>Laura2<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// =&gt; true</span></code></pre> </div> <div class="gatsby-highlight" data-language="tsx" style="display:flex;flex-direction:row;align-items:center;"><pre class="language-tsx" style="margin-right: 10px;font-size: 6pt;padding-bottom:20px"><code class="language-tsx" style="font-size: 9pt;"> <span class="token keyword">const</span> Laura1 <span class="token operator">=</span> <span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token string">"Laura"</span><span class="token punctuation">,</span> age<span class="token operator">:</span> <span class="token number">25</span> <span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token keyword">const</span> Laura2 <span class="token operator">=</span> <span class="token punctuation">{</span> age<span class="token operator">:</span> <span class="token number">25</span><span class="token punctuation">,</span> name<span class="token operator">:</span> <span class="token string">"Laura"</span> <span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token constant">JSON</span><span class="token punctuation">.</span><span class="token function">stringify</span><span class="token punctuation">(</span>Laura1<span class="token punctuation">)</span> <span class="token operator">===</span> <span class="token constant">JSON</span><span class="token punctuation">.</span><span class="token function">stringify</span><span class="token punctuation">(</span>Laura2<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// =&gt; false</span></code></pre> </div> </div> <p><br> We can use a deep equality. A <strong>deep equality is an equality based on what the object owns</strong>. Lodash provides <code>isEqual</code> to evaluate deep comparison.</p> <div class="gatsby-highlight" data-language="tsx"><pre class="language-tsx"><code class="language-tsx"><span class="token keyword">const</span> Laura1 <span class="token operator">=</span> <span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token string">"Laura"</span><span class="token punctuation">,</span> age<span class="token operator">:</span> <span class="token number">25</span> <span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token keyword">const</span> Laura2 <span class="token operator">=</span> <span class="token punctuation">{</span> age<span class="token operator">:</span> <span class="token number">25</span><span class="token punctuation">,</span> name<span class="token operator">:</span> <span class="token string">"Laura"</span> <span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token constant">JSON</span><span class="token punctuation">.</span><span class="token function">stringify</span><span class="token punctuation">(</span>Laura1<span class="token punctuation">)</span> <span class="token operator">===</span> <span class="token constant">JSON</span><span class="token punctuation">.</span><span class="token function">stringify</span><span class="token punctuation">(</span>Laura2<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// => false</span> <span class="token function">isEqual</span><span class="token punctuation">(</span>Laura1<span class="token punctuation">,</span> Laura2<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// => true</span></code></pre></div> <p><br> Therefore, the <code>isEqual</code> <strong>lodash function</strong> is a wiser choice to <strong>compare the props</strong>.</p> <div class="column-list"> <div> <div class="gatsby-highlight" data-language="tsx" style="margin-right: 10px;font-size: 6pt;"><pre class="language-tsx" style="padding: 20px;"><code class="language-tsx"><span class="token keyword">export</span> <span class="token keyword">default</span> <span class="token function">memo</span><span class="token punctuation">(</span>SidebarCard<span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre> </div> <span style="font-size:14px;color: #b3b3b3">Before custom comparison</span> </div> <div> <div class="gatsby-highlight" data-language="tsx" style="margin-right: 10px;font-size: 6pt;"><pre class="language-tsx" style="padding: 20px;"> <code class="language-tsx"> <span class="token keyword">const</span> <span class="token function-variable function">compareProps</span> <span class="token operator">=</span> <span class="token punctuation">(</span> propsBefore<span class="token operator">:</span> SidebarCardProps<span class="token punctuation">,</span> propsAfter<span class="token operator">:</span> SidebarCardProps <span class="token punctuation">)</span> <span class="token operator">=&gt;</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token function">isEqual</span><span class="token punctuation">(</span>propsAfter<span class="token punctuation">,</span> propsBefore<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token keyword">default</span> <span class="token function">memo</span><span class="token punctuation">(</span>SidebarCard<span class="token punctuation">,</span> compareProps<span class="token punctuation">)</span><span class="token punctuation">;</span> </code> </pre> </div> <span style="font-size:14px;color: #b3b3b3">With custom comparison</span> </div> </div> <p>Drum roll...</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/3a85870e0d34a5faeea9e6875cf62dfe/50383/profiler-not-rendered-card.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 43.78378378378379%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAYAAAAywQxIAAAACXBIWXMAABYlAAAWJQFJUiTwAAACDElEQVQoz5WQ204TURSG+yompu1MZ1o6M+20Mz0Bcrbg0Cmn1hSIigotkYOUloOcDIpGLAEJQb3SxzAaE1/rc88QL7j04su/9tp7/WvtFXCKw5SdMWYmykyXXaEulcn/RNTUpifou9dPwK0M4VaHmHg4zNRskZm5IrNPx29RW3CoeXqLEnPPXR8vnl9yGRztI5BYrpNYWiTZqGM117EF2a02uZ0toS2yQgv7u+SF5rfbPgUR59otUutrmGurZJob9BzsYUxVhOHhDnrnhMzXC6w3ByQ6bzHP32NefMC67lC4OmVwo0Hp7JDxy2Pc6xPczhHPWm1Ov3zm0/dvLLa3KD1eIDc0SkByymhXZ6R3WqjZHozmKvb5CeblR5LNNaz9bUIRnXi9Tvz4EKPzDnlzA3eyxp/ff/j54xe7r15jZwcwrV5hqCZIVStok1ME76qoZgbtaI/0yxXiow5qY5mwlkYeuI+6skJMfC3a2kRJ5nDcKka6h3zvCHZhkC4jSyAkxQlGk0h9w4QVg+CdCOqjJyiz88j9IyjiB1Iyi5TOo1RrRJtNYkt1QqJJUNSqcYuoZqOnCkItAp6JR0jWbjTUhSzMfSO7G7l7wJ8wrFuoYyVSLxpEig5hw0aKGMT0jF8nR02UrpSYUOznn6lv6J3FGjy86X1EzmsoKTqqnibywPUbhEU+ptviXrt5I/gLC05d9wDmRdYAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Profiler with not re-rendered third card" title="Profiler with not re-rendered third card" src="/static/3a85870e0d34a5faeea9e6875cf62dfe/50383/profiler-not-rendered-card.png" srcset="/static/3a85870e0d34a5faeea9e6875cf62dfe/1d79a/profiler-not-rendered-card.png 185w, /static/3a85870e0d34a5faeea9e6875cf62dfe/1efb2/profiler-not-rendered-card.png 370w, /static/3a85870e0d34a5faeea9e6875cf62dfe/50383/profiler-not-rendered-card.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>🥳 Yeah, got it! So for the 600 items list, when the second card is clicked on, here is the after memoization:</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/091903e2f2efbe6c8319f67287f4c94f/50383/profiler-memoized-600.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 40%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAYAAAD5nd/tAAAACXBIWXMAABYlAAAWJQFJUiTwAAAA+ElEQVQoz5WM4UvCUBTF3z/TtvfsbVJzrhRFP0SGWy5HESKVkKKgSJGR/vGn+66bzEqwDz/Ouefec0WcXuNhGOF+0CWN8fiSYDBK8DxO/8ckxdNrHyIcjlCfzlGbzFCfLdBYvKG5fEdrtUZ7tSHWx/G5QevjCyLodFEl/KsbBJ0IF7d3CGNDwpi5lqSsOZz3+pyHNF9mWo16EEr7cBwPtu3BsY26sCwN23IZkylZZt3tT/R+Tn3u0E5IfQ5JT+XpVlXmVeaV66PkBaQV9kZNx3i5u6lsO5QJY5zSGcNPs7nold7P/7rN9aiHh8pFfj38+fiQL85F8t03guYRCEhgCbUAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Profiler with not re-rendered card for 600 items" title="Profiler with not re-rendered card for 600 items" src="/static/091903e2f2efbe6c8319f67287f4c94f/50383/profiler-memoized-600.png" srcset="/static/091903e2f2efbe6c8319f67287f4c94f/1d79a/profiler-memoized-600.png 185w, /static/091903e2f2efbe6c8319f67287f4c94f/1efb2/profiler-memoized-600.png 370w, /static/091903e2f2efbe6c8319f67287f4c94f/50383/profiler-memoized-600.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h4>What if we only use memoize custom comparison without useCallback?</h4> <p>The <code>isEqual</code> <strong>on the <code>onSetSelected</code> function would have worked</strong>, the card wouldn’t have re-rendered. But it was a nice learning in case you have a function which is triggering a re-render.</p> <h3>Is there a difference between useMemo and useCallback?</h3> <p><code>useMemo</code> is usually used for variable memoization. But you can use a <code>useMemo</code> <strong>to memoize a function</strong>.</p> <blockquote> <p><a href="https://reactjs.org/docs/hooks-reference.html#useCallback">React documentation</a>:</p> <p><code>useCallback(fn, deps)</code> is equivalent to <code>useMemo(() => fn, deps)</code>.</p> </blockquote> <h3>Why not always use memoization?</h3> <p>Memoizing is not free, it costs memory and the comparison cost. It adds complexity to your code so it will need more efforts to read/to refactor.</p> <h2>Conclusion</h2> <p>Thanks to React.memo() HOC and memoization hooks, useless re-renders were avoided! I hope you've learned a bunch of things!</p> <h3>Go further:</h3> <p><a href="https://blog.theodo.com/2019/05/fantastic-hooks/">React: Fantastic Hooks and How to Use Them</a></p> <p><a href="https://blog.theodo.com/2018/09/use-react-virtualized/">React-Virtualized: Why, When and How you should use it</a></p> <p><a href="https://twitter.com/iamakulov/status/1362397512569651200">Ivan Akulov’s thread on re-renders</a></p> <p><a href="https://dmitripavlutin.com/use-react-memo-wisely/">Use React.memo() wisely</a></p> <p><a href="https://dmitripavlutin.com/react-useref-guide/">The Complete Guide to useRef() and Refs in React</a></p><![CDATA[Mastering React-admin Resources to Improve Your App Performance]]>/2022/01/react-admin-resources-improve-perf//2022/01/react-admin-resources-improve-perf/Tue, 18 Jan 2022 00:00:00 GMT<p>Let me tell you the story of how I improved my <strong>react-admin</strong> app runtime loading performance from <strong>1 minute to 1 second</strong>.</p> <p>I work at <a href="https://www.theodo.fr/">Theodo</a> on a project with many websites. To configure and manage all of them, we had an administration tool (let's call it AdminV1). But AdminV1 was old: it was an Angular project coded with <a href="https://coffeescript.org/">coffee.script</a> and <a href="https://jade-lang.com/">jade</a>, and was not very maintainable. And the product was difficult to use: there were too many menus, UX was bad, it was difficult to onboard new administrators on the application.</p> <p>The team decided to create a brand new application with modern technologies: AdminV2, with <a href="https://marmelab.com/react-admin/">react-admin</a>, a framework for building administration applications.</p> <p>Everybody was happy: for developers, it was easier to implement new features. With a few lines of code, they could create tables to administrate users, app configurations and privileges. For Product Owners and administrators, the design was more ergonomic and it was easier to manage their applications.</p> <p>We migrated many features from AdminV1 to AdminV2 and added new administration tools. Sponsors were delighted. But sprint after sprint, AdminV2 became slower to start. After some months, users had to wait <strong>more than one minute</strong> to access the app. It was painful to develop new features on AdminV2 because each hot reload took 1 minute, and users were frustrated by waiting so long to access the application.</p> <p>We started to investigate and found that by deleting some <strong>Resources</strong> in our application, the speed at runtime increased significantly</p> <h2>But what are react-admin Resources ?</h2> <p><em>"<code>&#x3C;Resource></code> components are fundamental building blocks in react-admin apps. They are strings that refer to an entity type."</em>, according to <a href="https://marmelab.com/react-admin/Resource.html">react-admin documentation</a>.</p> <p>To illustrate, on my project, we need to administrate users of websites. We have to create a resource "/user", associated with components:</p> <ul> <li>UserList (for table view)</li> <li>UserEdit (to edit an user)</li> <li>UserCreate (to create a new user).</li> </ul> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">import</span> UsersList <span class="token keyword">from</span> <span class="token string">"components/Users/list"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> UsersEdit <span class="token keyword">from</span> <span class="token string">"components/Users/edit"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> UsersCreate <span class="token keyword">from</span> <span class="token string">"components/Users/create"</span><span class="token punctuation">;</span> <span class="token keyword">const</span> <span class="token function-variable function">UserResource</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">(</span> <span class="token operator">&lt;</span>Resource key<span class="token operator">=</span><span class="token string">"/user"</span> name<span class="token operator">=</span><span class="token string">"/user"</span> <span class="token comment">// endpoint of the API to call to get users</span> create<span class="token operator">=</span><span class="token punctuation">{</span>UsersCreate<span class="token punctuation">}</span> list<span class="token operator">=</span><span class="token punctuation">{</span>UsersList<span class="token punctuation">}</span> edit<span class="token operator">=</span><span class="token punctuation">{</span>UsersEdit<span class="token punctuation">}</span> <span class="token operator">/</span><span class="token operator">></span> <span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre></div> <p>React-admin is in charge of <strong>storing</strong> this information. When the user goes on /user URL, react-admin <strong>displays</strong> the list, edit or create view and <strong>manages</strong> all the logic to get, edit and create users.</p> <p>Sounds like magic, not helping us understand why creating many Resources degrades the performance.</p> <p>I dived deeper into the react-admin code to understand how the framework creates resources and links them with components.</p> <p>A <code>&#x3C;Resource /></code> component is declared like this:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">const</span> <span class="token function-variable function">Resource</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">props<span class="token operator">:</span> ResourceProps</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> <span class="token punctuation">{</span> intent <span class="token operator">=</span> <span class="token string">"route"</span><span class="token punctuation">,</span> <span class="token operator">...</span>rest <span class="token punctuation">}</span> <span class="token operator">=</span> props<span class="token punctuation">;</span> <span class="token keyword">return</span> intent <span class="token operator">===</span> <span class="token string">"registration"</span> <span class="token operator">?</span> <span class="token punctuation">(</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">ResourceRegister</span></span> <span class="token spread"><span class="token punctuation">{</span><span class="token punctuation">...</span><span class="token attr-value">rest</span><span class="token punctuation">}</span></span> <span class="token punctuation">/></span></span> <span class="token punctuation">)</span> <span class="token operator">:</span> <span class="token punctuation">(</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">ResourceRoutes</span></span> <span class="token spread"><span class="token punctuation">{</span><span class="token punctuation">...</span><span class="token attr-value">rest</span><span class="token punctuation">}</span></span> <span class="token punctuation">/></span></span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre></div> <blockquote> <blockquote> <p><em>React-admin <a href="https://github.com/marmelab/react-admin">Code</a></em></p> </blockquote> </blockquote> <p>Your intent can be either <code>registration</code> or <code>route</code>:</p> <ul> <li> <p><code>ResourceRegister</code>'s purpose is to dispatch an action that <strong>saves your Resource into a redux store</strong>. Each time you define a <code>&#x3C;Resource /></code> component in your app, react-admin will register your Resource's props into the redux store:</p> <blockquote> <ul> <li>the <strong>name</strong>, which is the URL of the resource and also the API endpoint to fetch your entity data</li> <li>which <strong>views</strong> are available to manage the entity. For our user entity, available views are list, edit and create</li> </ul> </blockquote> </li> </ul> <div class="gatsby-highlight" data-language="js"><pre class="language-js"><code class="language-js"><span class="token function">dispatch</span><span class="token punctuation">(</span> <span class="token function">registerResource</span><span class="token punctuation">(</span><span class="token punctuation">{</span> name<span class="token punctuation">,</span> hasList<span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span> hasEdit<span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span> hasShow<span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span> hasCreate<span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token punctuation">)</span> <span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre></div> <blockquote> <blockquote> <p><em>React-admin <a href="https://github.com/marmelab/react-admin">Code</a></em></p> </blockquote> </blockquote> <ul> <li><code>ResourceRoutes</code> is rendered when a user wants to <strong>access</strong> the URL associated with the Resource. The URL (which is also generally the endpoint of the API) is stored in the Resource's <code>name</code> prop. ResourceRoutes renders all <code>&#x3C;Route /></code> components associated with each view defined in the Resource. For our Entity "user", it will be:</li> </ul> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">const</span> <span class="token function-variable function">ResourceRoutes</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">(</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">ResourceContextProvider</span></span> <span class="token attr-name">value</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>name<span class="token punctuation">}</span></span><span class="token punctuation">></span></span><span class="token plain-text"> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">Switch</span></span><span class="token punctuation">></span></span><span class="token plain-text"> &lt;Route path=</span><span class="token punctuation">{</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>basePath<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">/create</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">}</span><span class="token plain-text"> // Create view render=</span><span class="token punctuation">{</span><span class="token punctuation">(</span><span class="token parameter">routeProps</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">(</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">WithPermissions</span></span> <span class="token attr-name">component</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>CreateUser<span class="token punctuation">}</span></span> <span class="token spread"><span class="token punctuation">{</span><span class="token punctuation">...</span><span class="token attr-value">routeProps</span><span class="token punctuation">}</span></span> <span class="token punctuation">/></span></span> <span class="token punctuation">)</span><span class="token punctuation">}</span><span class="token plain-text"> /> &lt;Route path=</span><span class="token punctuation">{</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>basePath<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">/:id</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">}</span><span class="token plain-text"> // Edit view render=</span><span class="token punctuation">{</span><span class="token punctuation">(</span><span class="token parameter">routeProps</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">(</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">WithPermissions</span></span> <span class="token attr-name">component</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>EditUser<span class="token punctuation">}</span></span> <span class="token spread"><span class="token punctuation">{</span><span class="token punctuation">...</span><span class="token attr-value">routeProps</span><span class="token punctuation">}</span></span> <span class="token punctuation">/></span></span> <span class="token punctuation">)</span><span class="token punctuation">}</span><span class="token plain-text"> /> &lt;Route path=</span><span class="token punctuation">{</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>basePath<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">}</span><span class="token plain-text"> // ListView render=</span><span class="token punctuation">{</span><span class="token punctuation">(</span><span class="token parameter">routeProps</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">(</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">WithPermissions</span></span> <span class="token attr-name">component</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>ListUser<span class="token punctuation">}</span></span> <span class="token spread"><span class="token punctuation">{</span><span class="token punctuation">...</span><span class="token attr-value">routeProps</span><span class="token punctuation">}</span></span> <span class="token punctuation">/></span></span> <span class="token punctuation">)</span><span class="token punctuation">}</span><span class="token plain-text"> /> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span><span class="token class-name">Switch</span></span><span class="token punctuation">></span></span><span class="token plain-text"> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span><span class="token class-name">ResourceContextProvider</span></span><span class="token punctuation">></span></span> <span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre></div> <blockquote> <blockquote> <p><em>React-admin <a href="https://github.com/marmelab/react-admin">Code</a></em></p> </blockquote> </blockquote> <p>For example, if an administrator connects to a react-admin app and wants to see a list of users, the framework will load resources like this:</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/1ced9204b0808fe072178c9916490f69/50383/react-admin-resources-scheme.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 45.4054054054054%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAYAAAAywQxIAAAACXBIWXMAAAsSAAALEgHS3X78AAABTElEQVQoz41S2U7DQAzM/38FQrwh+BIQQqAWUClJlebe+0p4GbybKpRL9MGyvWuPZz2bvRUT1s8T3BSg7AhpA4TxZGGOl9wvd9H/Zdl2u8PTaoNeWZTNkBqMD9DWgwmdvPXjSWAJsCOw7vYRJVPY5CW4dpDagEuNuu0xcAljHZQ7EVAqCmRIDcpGP4Iph33H0DACHQT40XP/fbLyHjr4pThaBK0I8PLqGsW+gfbT1x1+G3A8KGv6EWU9UtMsxAw4oe45zs4vEqAJ7z/Y/cY09mcvrxz3DwPsNELouSgyjCyqXhwxCyTWCK5MEkpTTbwbSDhOgkaLeVatVyjvbpC3HHlZo+WKRLEkhkDVdLCO1CaQZpAQUmGbF9hXtAbj0JKQTdej2JWoajpzUZRKQhYcjArYYUpUOk0kL4z73G38UtQUTdrDGbHvmEz1Mf4A/sOspJgtIzkAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="React-admin Resources schema" title="React-admin Resources schema" src="/static/1ced9204b0808fe072178c9916490f69/50383/react-admin-resources-scheme.png" srcset="/static/1ced9204b0808fe072178c9916490f69/1d79a/react-admin-resources-scheme.png 185w, /static/1ced9204b0808fe072178c9916490f69/1efb2/react-admin-resources-scheme.png 370w, /static/1ced9204b0808fe072178c9916490f69/50383/react-admin-resources-scheme.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>This allows calling the API only when the user goes on the associated Menu and to not call all the endpoints available in the application at runtime.</p> <h2>Ok, so what's going on with AdminV2 ?</h2> <p>In AdminV2, we defined our Resources like this:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">const</span> <span class="token function-variable function">RootComponent</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> resources <span class="token operator">=</span> <span class="token function">getResources</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token keyword">return</span> <span class="token punctuation">(</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">AdminContext</span></span> <span class="token spread"><span class="token punctuation">{</span><span class="token punctuation">...</span><span class="token attr-value">props</span><span class="token punctuation">}</span></span><span class="token punctuation">></span></span><span class="token plain-text"> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">AdminUI</span></span><span class="token punctuation">></span></span><span class="token plain-text"> </span><span class="token punctuation">{</span>resources<span class="token punctuation">}</span><span class="token plain-text"> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span><span class="token class-name">AdminUI</span></span><span class="token punctuation">></span></span><span class="token plain-text"> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">AdminContext</span></span> <span class="token punctuation">/></span></span><span class="token plain-text"> }</span></code></pre></div> <p><code>GetResources</code> returned <strong>all Resources</strong> in our application: user Resources and other <strong>600 Resources</strong> (yes that's a lot).</p> <p><strong>So what's happening when a user access AdminV2 ?</strong></p> <ol> <li>First, over 600 <code>&#x3C;Resources /></code> components are rendered</li> <li>Then, each of these <code>&#x3C;Resources /></code> dispatches an action to store their data in redux store (the "registration" intent seen above)</li> <li>React-admin redux reducer gets these <strong>600 actions</strong> and for each of them, it <strong>stores the data</strong> in the redux store</li> <li>Finally, when the reducer has updated the store with all resources action payload, the home page of AdminV2 is displayed</li> </ol> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 242px; " > <a class="gatsby-resp-image-link" href="/static/67b5a4b5062595041297ecf93cc0364f/a1a85/adminv2-runtime-steps.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 340.5405405405405%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAABECAYAAACBBxX6AAAACXBIWXMAAAsSAAALEgHS3X78AAAH/klEQVRYw6VYS4/kVhXuf8KWHSs2sIjEYnZBYgUSkViwyIpIIUERCxhFoyQKmiAQRIkQ4q1EQkIoiRQNmQwznZnuqX5Ud3W93+93ufwo21Wusl1fzrllu+x69NDB0unruvY9Pufec77znT5QjRl2iaJPgzEs4We75GDfA3/hZLaAbjlC+P46Zdcq5IXadI7eYIC//+oV/OPuq+j1+zS3uLmFwr2JCWMBXB0/wKM/3cGDP9xGMvYQpgPIE2Ov0v0WkkJ9vkTm6hjj8iGG+fvIJmPiI/zs5hayy7RnvVQN5d/8BZXf/hW9ZA2atYgc0P+sUDUtKNIE8hv3If/0E8gvfwzlrQdQhhrUqXVzhYpwmxYpBsaqDkmZiHueU64JnYNdcSdPpt5I+0guFhsdpMsNcc9zrFTeE5cHW3FHrk7nFHe0f4ZlY7Zw0eiOUG31Qbdi3pzbQjRz2/WIhROKu1Z/hJNEBsVaC6l8BblKAzmy7uQyjVOazxRriKfyOLvKot2XxJodFk4DhY3uQCxOF6qI0ciLU4UKnsaTiF2kxEcS2aJQ2B3JISune1wmpSa5zKNBe8Yu1zpDlJs90LQ3b8Mg131jtlyOHEh4s+kAprRX56kCTpN5ca/opqdkN2AchCcm5gwL24HjuHBdF467Gm2aY3Hd6Dy/y2vCOgILNXrgukvc9HKXS7E2sNBXaC1s8YLjOIgnUqjU6HSLZVylc+gPR2i2uyiUq2h3+8jkS5AVNVA6X9gbLpOwm+KL5EqpWhcKas02srRY1Sbo9PokA9Rp7knsHKOxLN5fkoW8BRGFilBoY+mSlcuVYrYuky8SBg5RrTfFyB8ZSTLGsiI88a+IQv7D6WStn8OyLFQbLaGwWKkJF2v0m6VLilnhdDZbvz9fiOhYKaQwUGYOxtUU1I/fxKwav9Gh8Fb5IeRZaEHWdPR+/wP073wLg18/D8uYgI/IdlfCixw/fJy1zMgydSMWhctjcrn7xxcx+MU30P3d9zEsxCHXMhjXc1D7TcpvCXXKFk4zgTYeEvmotBHYFPlTG8NOE53DDzCo5aH026QwDblZhD7qolgn+Co1YFr2ThyMWMhgyWarXM2oXqi0n4zWPComldA5hVGji2y5KXJb3qjTWxaqegiBRTUz1y8zwM7mK4WVpsBC/zTZfQaKLYCNfGHDHZ5jq/JVwkbCQQZYP8yGsibgayfAhifDrvBCVhjPlBBLZAVG5glw//PoGB/++1Pc/yJG+9sWUCbvQhthEbnIgcr5ycI5zqhiTGe4SGbR7PTwmFLv6PQCcf7dGwaoHUKblcK5BxA3uWzH3S4BPDEl1/zIH47G0A1TpJc20QPh34qqwaBni8VCAANfU2uxDV+2l+xzcvfp2YVAFB6PTs4FbBVKKzk8PiGXz9Ai1wO0cdxNxPbgi5Bm6aENo4k5JRQnyxj7TNrDiW5Q+tkBcvsW+mijBBbSaS5sN9iXpbu6Z/di55e4TGZwmcrgJJ4Qc75lvkI+tGhgm3PxdauRgGuZ3gI3Ysl6XEaU8aWHCn6gsP+v1wltvo3xB69GXr62ntBHjBBxWoUNaR/LKnrvPA/p7i303nhOoIyuqdBkSYyyZkBSJ9BpMVc5X/yyu5HLlLtTB72zT9F5/0foEeJIDF3px5AKZ9DaJYE2GUIbkRGTUK7rs41cZrTRPSoycyEZNmQa1dBXDR8cPLRRnok2ATCYgSihvGaEYRqSq7TE/aqGr5BGfxbaBA/0NdVgqzLEvhK5iuA2vGcjRUd7IBGxooKlmXss3NHkCLQhqxL5Ks6I38RTOUHnPvnsEH/750e49/AIhaoHvJMdaMNWcV4uQsWI7zmKGIGyVKv7QwnxqzQSxCgyhTI6w3FA6SLchkdeTCEdBK4fj5uBHCmjFOjhyhco5JL4VS9eGwKHlUKfovUGQwxGkiBIQ4lgjFKSwYGpCHMc06RDoXlmD/zeXm4jeCB9iZnX54+e4JxGhq5GqyMY2BOCrEQqK1gZjywMHNZ8LvY46jKjjRNCm+XNeOKcIE0Og4PKNYEqmPzfI8wlOYCw8OHsEv/jky3CSVL8+dtI3XoBpZduwwnVlrCtmwo5rLYpsUCbCdI/fAn57/4YKZJBtgSFGh251YNGRGpEbVlfUkVTpFGsalzo95J2blWJ3beO4si+9hYa9w4hETHqnybRv8xCI0VFyuV0se5lhLnVjYZTds1tKFFlDlKiBytuQ/tK+MdKNtFmzRSu4zZsqWaKcc1tpmtus1Phdle6E23CAMEKy80VWYri4W6le9FG9eCdu6ezZAEnVznRpfIeqtcB7ObEjBTYHgUWHRWFB9dk5jauV9QFPabRone3CGd4gl/eF8C74pCjlNdouwD2/0Eba25vow27M6dET+cKGBDScE/CaMJMoUydVbvbE93UkOaK5Rr1KwOBQAvKKl673UlRDBrEZWJEN1LZAp4Q0jCiMFHKEjIzCnF3lS9VRHv2xfEpHj89E03SMqzQp7ius/zqLi9CaMM3BnUARXLpvYf3UOp3A0blPgNpfDqydShs4ffefRtfe+1FfOfubap0s62emEmSGH2h38x4tc2w0ag1G6k6br3zOr7+y5/gm3d+hkyTGkU6mEy7LSwXzZHfJXgS/v9OJLDZZZNqb6xYxCsf/hn3k1cYEFwVSNFhPoeLej1ItWjd3tNJBRx7Rv0y4eqM+ly2mrGPP6Rv/F9mE2n2pp5Mbo1FQ2juJUO75jflS+HFKtKjhtUhAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Old AdminV2 Resources rendering" title="Old AdminV2 Resources rendering" src="/static/67b5a4b5062595041297ecf93cc0364f/a1a85/adminv2-runtime-steps.png" srcset="/static/67b5a4b5062595041297ecf93cc0364f/1d79a/adminv2-runtime-steps.png 185w, /static/67b5a4b5062595041297ecf93cc0364f/a1a85/adminv2-runtime-steps.png 242w" sizes="(max-width: 242px) 100vw, 242px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>These 4 steps take more than a minute. Eventually, the user has gone for a coffee, otherwise, he left the app and will never come back again</p> <h2>How to avoid this 1 minute of loading time?</h2> <p>We thought about diverse solutions:</p> <h3>1 - Load only resources to which the user has access</h3> <p>On AdminV2, to access each menu, the user needs to have the associated privilege. If the user has access to <strong>one menu</strong>, we don't have to load all resources, but <strong>only the resources used in the menu</strong>.</p> <p>This solves the problem for regular users with few privileges. But Developers and Product Owners have access to all menus.</p> <h3>2 - Patch react-admin to modify Resource component</h3> <p>We thought of modifying directly the react-admin Resource component, to <strong>modify the dispatch action</strong>. Instead of dispatching one action for each Resource, we could dispatch <strong>one action for multiple resources</strong>. This way, we don't have to update the state 600 times.</p> <p>But this solution is not very maintainable: it implies updating the patch each time we update react-admin.</p> <h3>3 - Lazy load Resources</h3> <p>We don't have to create every Resources at the initialization of the application. An administrator will never access all AdminV2 menus and, even if he wanted to, in most cases, he doesn't have all menus accesses. We need to <strong>load Resources only when the user needs them</strong>. So we decided to create them <strong>when the user accesses the associated url</strong>. We modified our <code>RootComponent</code> like this:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">const</span> <span class="token function-variable function">RootComponent</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> <span class="token punctuation">[</span>resources<span class="token punctuation">,</span> setResources<span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token function">useState</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> location <span class="token operator">=</span> <span class="token function">useLocation</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">useEffect</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> resourcesToAdd <span class="token operator">=</span> <span class="token function">getResources</span><span class="token punctuation">(</span>location<span class="token punctuation">.</span>pathname<span class="token punctuation">)</span> <span class="token function">setResources</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token operator">...</span>resources<span class="token punctuation">,</span> <span class="token operator">...</span>resourcesToAdd<span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">[</span>location<span class="token punctuation">.</span>pathname<span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token keyword">return</span> <span class="token punctuation">(</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">AdminContext</span></span> <span class="token spread"><span class="token punctuation">{</span><span class="token punctuation">...</span><span class="token attr-value">props</span><span class="token punctuation">}</span></span><span class="token punctuation">></span></span><span class="token plain-text"> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">AdminUI</span></span><span class="token punctuation">></span></span><span class="token plain-text"> </span><span class="token punctuation">{</span>resources<span class="token punctuation">}</span><span class="token plain-text"> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span><span class="token class-name">AdminUI</span></span><span class="token punctuation">></span></span><span class="token plain-text"> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">AdminContext</span></span> <span class="token punctuation">/></span></span><span class="token plain-text"> ) }</span></code></pre></div> <p>We modified the getResource function so that it takes the <strong>current URL</strong> in parameters and returned the <strong>Resources associated with the URL</strong>.</p> <p>For example, if an administrator wants to access the user administration menu (URL '/user'), <code>getResources</code> will only return</p> <blockquote> <p><code>&#x3C;Resource name="/user" list={UserList} edit={UserEdit} create={UserCreate} />.</code></p> </blockquote> <p>The <code>RootComponent</code> will render the new Resource:</p> <ul> <li>The first intent will be <code>registration</code> and the Resource will be <strong>stored in redux state</strong></li> <li>The second intent will be <code>route</code>. React-admin will <strong>create the Routes</strong> for user entities and the administrator will access the user management menu</li> </ul> <p>In order not to save resources each time the user goes back on the same URL, we added a filter:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">const</span> <span class="token function-variable function">RootComponent</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> <span class="token punctuation">[</span>resources<span class="token punctuation">,</span> setResources<span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token function">useState</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> <span class="token punctuation">[</span>loadedResources<span class="token punctuation">,</span> setLoadedResources<span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token function">useRef</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> location <span class="token operator">=</span> <span class="token function">useLocation</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">useEffect</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> resourcesToAdd <span class="token operator">=</span> <span class="token function">getResources</span><span class="token punctuation">(</span>location<span class="token punctuation">.</span>pathname<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">filter</span><span class="token punctuation">(</span> <span class="token punctuation">(</span><span class="token parameter">resource</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token operator">!</span>loadedResources<span class="token punctuation">.</span><span class="token function">includes</span><span class="token punctuation">(</span>resource<span class="token punctuation">.</span>name<span class="token punctuation">)</span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">setResources</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token operator">...</span>resources<span class="token punctuation">,</span> <span class="token operator">...</span>resourcesToAdd<span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">setLoadedResources</span><span class="token punctuation">(</span><span class="token punctuation">[</span> <span class="token operator">...</span>loadedResources<span class="token punctuation">,</span> <span class="token operator">...</span>resourcesToAdd<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter">resource</span><span class="token punctuation">)</span> <span class="token operator">=></span> resource<span class="token punctuation">.</span>name<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">[</span>location<span class="token punctuation">.</span>pathname<span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">AdminUI</span></span><span class="token punctuation">></span></span><span class="token punctuation">{</span>resources<span class="token punctuation">}</span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span><span class="token class-name">AdminUI</span></span><span class="token punctuation">></span></span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre></div> <h2>Let's sum up</h2> <p>Before, the user had to wait <strong>more than one minute</strong> to access the app. Then the navigation was quite quick.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/b930dba388b85b764356274dd2696df1/50383/adminv2-old-process.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 36.21621621621621%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAHCAYAAAAIy204AAAACXBIWXMAAAsSAAALEgHS3X78AAABWElEQVQoz0VRW3LCMAzM/a/EDfpRhqEzLc8SIEASx884dlKY7dot7YfGklbaZFfFZq5QrizcGKH7wBig3JDz9D5zYXoI7aCfte7RsafyXPh9BxTH/QXH/RmSjVYaAhEufsH2Hm4IMMOEnnUjJGqhYOMEP06o2w51pxCI+SGi56z1AUVXVqi3Ja4kOzcK7ZXk521e8Fw29QnqdoLUFvH+QMNZW7cI45iJlmWJRkp4P8AlQm0p00UYFibcoeoK8rThn43QfiL5DpqExo9QJGjXB6hLw9kRhqrmhwMqqdCHKVtRrF4lPt919jD7RYk2Pih3ZNDXQPn8UMqtZ02JhssJt+wNxE0f//wuzh8bHNe7LLmqBaPLEkQnoZSGcZ5e/dTWOijrIYm3rYAknnx7Hi4dtFAVgZvMR5EcTuBsuUAluuxJ6s3eFmjSMheSFd3qBeJ6JEHMsv8JA74B6HMT8+Ak/5IAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Old process to access User Menu on Admin V2" title="Old process to access User Menu on Admin V2" src="/static/b930dba388b85b764356274dd2696df1/50383/adminv2-old-process.png" srcset="/static/b930dba388b85b764356274dd2696df1/1d79a/adminv2-old-process.png 185w, /static/b930dba388b85b764356274dd2696df1/1efb2/adminv2-old-process.png 370w, /static/b930dba388b85b764356274dd2696df1/50383/adminv2-old-process.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>Now, the user access AdminV2 in <strong>one second</strong>. And the navigation stays quite quick!</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/361f4a48ed0dbd5f8f5292dac03a5c6a/50383/adminv2-new-process.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 37.83783783783784%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAYAAAD5nd/tAAAACXBIWXMAAAsSAAALEgHS3X78AAABWUlEQVQoz11R2W7CMBDM//9YVfUJnsqhtoGSy4nP2CS00nS8gYJ4sJzJjmd3doqmjgjTBDue5fi4fJuQ/u/nb3Pl3nn5/4KL07FB1Vu0g4UPEYO2sH6kMAkuLJgPYprgfIBhzVEkEYeYhPfYsNC9FXBUA972O2jrkM4XrMoSH3WNef5BT+GX7QYhJcFr1vZ1hfN5hntyU7RtouVZQMeHPpFE250N6DlNJg4+ojEempx8ck258W75tpYsWH93fBihTIAmqadFQ+HcOU+QLTtaHtlIJmKTQLs2POz3OqHssGsGdOx+6jTa3sieRgrUg8Hq61Pw5fIrNr/aRkSzYA7PxWs4caZoXCa0UpikUxbptWE4IwZOuzs1aLRDpz3KRlFQiZMD71ophuYZUISuD1Cql5AKc93LmC54ZSjbU8XCJMneGuWTV2DDYn1dHrCpKkSGl1PX5TuHCML/A+EyYZh+UyZhAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="New process to access User Menu on Admin V2" title="New process to access User Menu on Admin V2" src="/static/361f4a48ed0dbd5f8f5292dac03a5c6a/50383/adminv2-new-process.png" srcset="/static/361f4a48ed0dbd5f8f5292dac03a5c6a/1d79a/adminv2-new-process.png 185w, /static/361f4a48ed0dbd5f8f5292dac03a5c6a/1efb2/adminv2-new-process.png 370w, /static/361f4a48ed0dbd5f8f5292dac03a5c6a/50383/adminv2-new-process.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>As developers, when using "black box" framework like react-admin, we do not always try to understand how it really works. By taking the time to explore the source code of these frameworks, by understanding what libraries they use, how they are implemented, we can <strong>save our time later</strong> and help us detect this kind of performance issue sooner.</p> <p>That is the conclusion of AdminV2 performance story. And developers developed happily ever after and added a lot of features to AdminV2!</p><![CDATA[TypeScript: How Type Guards Can Help You Get Rid of 'as']]>/2022/01/typescript-replace-as-typeguards//2022/01/typescript-replace-as-typeguards/Wed, 12 Jan 2022 00:00:00 GMT<p>In TypeScript (TS), <strong>the "as" keyword</strong> is widely used when manipulating types. However, you should employ it with caution, as it does not provide any guarantee on the real <strong>types</strong> of your objects and could generate unexpected bugs. In this article, I will go into more detail about “as” and explain how to avoid it by using <strong>type guards</strong>, which represent a safer alternative.</p> <p>Before I started working at Theodo Lyon, I had only used Vanilla JavaScript and HTML in my previous web development experiences. So when I discovered TypeScript (TS) earlier this year, it seemed like a miraculous solution for writing safer code! It was when I joined a team working on a web app using <a href="https://reactjs.org/">React</a> jointly with <a href="https://www.typescriptlang.org/">TypeScript</a>. React is a free and open-source "JavaScript library for building user interfaces" developed by Facebook, while Typescript is a programming language developed as a superset of JavaScript that adds optional static typing to it. In other words, defined variables are assigned a specific and permanent type at creation, hence securing their future use.</p> <p>A few months ago, we started noticing many obscure bug occurrences with no obvious explanation. Also, the investigation was not satisfactory due to the lack of error logs: usually, the user would simply land on a blank page instead of the desired one.</p> <p>One of the most common causes was that some objects suddenly became <code>undefined</code> while that was not an option for TS since they had different types. For example, we encountered error messages such as :</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/0df7b908f4aeab0e2d0282f216f712e4/50383/undefined-error.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 7.567567567567568%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAACCAIAAADXZGvcAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAaElEQVQI1xWKSQ6AIBAE/f/vjEZE3BAEcUn0IMOAB4dLd1eli+R2EAO0AhedB+twkpTAha9ZGGfylL5qCIH3hG9ZA+Oo1uK7HyrUJh1XNBalimYLo8TVko/WpeMkk7c2qPItDHO+be4H341tO9N5afYAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="TypeError: Cannot read properties of undefined (reading &#39;map&#39;)" title="TypeError: Cannot read properties of undefined (reading &#39;map&#39;)" src="/static/0df7b908f4aeab0e2d0282f216f712e4/50383/undefined-error.png" srcset="/static/0df7b908f4aeab0e2d0282f216f712e4/1d79a/undefined-error.png 185w, /static/0df7b908f4aeab0e2d0282f216f712e4/1efb2/undefined-error.png 370w, /static/0df7b908f4aeab0e2d0282f216f712e4/50383/undefined-error.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>However, most of these occurrences would not even leave any error message and could lie undetected for a long time. Hence the investigation was even more painful!</p> <p>Through our analysis, we found out that most of these errors were linked to the usage of the <code>as</code> keyword in TS.</p> <p>For example, suppose we have an API used to retrieve books data, and a type <code>Book</code> defined as:</p> <div class="gatsby-highlight" data-language="typescript"><pre class="language-typescript"><code class="language-typescript"><span class="token keyword">export</span> <span class="token keyword">interface</span> <span class="token class-name">Book</span> <span class="token punctuation">{</span> id<span class="token operator">:</span> <span class="token builtin">number</span><span class="token punctuation">;</span> author<span class="token operator">:</span> <span class="token builtin">string</span><span class="token punctuation">;</span> publisher<span class="token operator">?</span><span class="token operator">:</span> <span class="token builtin">string</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></code></pre></div> <p>When I fetch the book of id 5:</p> <div class="gatsby-highlight" data-language="typescript"><pre class="language-typescript"><code class="language-typescript"><span class="token keyword">const</span> book <span class="token operator">=</span> <span class="token function">fetch</span><span class="token punctuation">(</span><span class="token string">"baseUrl/get-book?id=5"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre></div> <p>the received object will be of type <code>unknown</code>. But since I know the expected return type, I can force TS to use the type <code>Book</code> by using <code>as</code>:</p> <div class="gatsby-highlight" data-language="typescript"><pre class="language-typescript"><code class="language-typescript"><span class="token keyword">const</span> book <span class="token operator">=</span> <span class="token function">fetch</span><span class="token punctuation">(</span><span class="token string">"baseUrl/get-book?id=5"</span><span class="token punctuation">)</span> <span class="token keyword">as</span> Book<span class="token punctuation">;</span></code></pre></div> <h3>But how does "as" work exactly?</h3> <p>This keyword is a <a href="https://www.typescriptlang.org/docs/handbook/2/everyday-types.html#type-assertions">Type Assertion</a> used to inform the compiler to consider an object as a different type from the one inferred. The <code>as</code> assertion does not perform any transformation on the data stored in the variable and is removed at compilation. TS even warns us about this in the <a href="https://www.typescriptlang.org/docs/handbook/2/everyday-types.html#type-assertions">official documentation</a>: <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <span class="gatsby-resp-image-background-image" style="padding-bottom: 8.64864864864865%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAACCAIAAADXZGvcAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAUUlEQVQI1z2LQQ6AIAwE+f8nKYVCgrQiN/XiBtRk0sxuuo6oEW0hNMBsU3Q1zCrScx4p9ZT2hciBEreU4fDhfY3RIP+MqCICLM1u1fPjmrzyAMO9bRVdkCdLAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Reminder: Because type assertions are removed at compile-time, there is no runtime checking associated with a type assertion. There won’t be an exception or null generated if the type assertion is wrong." title="Reminder: Because type assertions are removed at compile-time, there is no runtime checking associated with a type assertion. There won’t be an exception or null generated if the type assertion is wrong." src="/static/c4d8603d28f6695c3cf91bf3dd137af6/50383/no-runtime-checking.png" srcset="/static/c4d8603d28f6695c3cf91bf3dd137af6/1d79a/no-runtime-checking.png 185w, /static/c4d8603d28f6695c3cf91bf3dd137af6/1efb2/no-runtime-checking.png 370w, /static/c4d8603d28f6695c3cf91bf3dd137af6/50383/no-runtime-checking.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </span></p> <p>Furthermore, it only checks that the assertion is feasible. For example, you can not use it to turn a string into a number: "TypeScript only allows type assertions which convert to a more specific or a less specific version of a type." <em>(<a href="https://www.typescriptlang.org/docs/handbook/2/everyday-types.html">"TypeScript Handbook"</a>)</em></p> <p>By digging deeper in the source code, the <code>as</code> does not perform any checks on the object but only serves as a static typing tool:</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/3c9ef97c57309804760e4a16cc069a83/50383/source-code-ts.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 8.64864864864865%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAACCAIAAADXZGvcAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAZ0lEQVQI11XLuwqAMAxAUf//y1zcpVRp0kF8NUkbX1VBwcnlwB1ucS2Y1fFkut4OWE22lOBSaPdoVRwRRgYmXKVRaYm8EIaAkUAIinc+EmzSJDYymtjX8wyJ3b1AVjzUZ/Wvp/7y8wEy/G6UY1/TMwAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Source code of TypeScript showing the code of &quot;as&quot;" title="Source code of TypeScript showing the code of &quot;as&quot;" src="/static/3c9ef97c57309804760e4a16cc069a83/50383/source-code-ts.png" srcset="/static/3c9ef97c57309804760e4a16cc069a83/1d79a/source-code-ts.png 185w, /static/3c9ef97c57309804760e4a16cc069a83/1efb2/source-code-ts.png 370w, /static/3c9ef97c57309804760e4a16cc069a83/50383/source-code-ts.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span> <em>Screenshot extracted from source code <code>typescript.js</code></em></p> <p>Usually, we used to employ the <code>as</code> keyword in order to handle data coming from external sources. Examples of such sources could be APIs or imported excel files. Since the content was not known in advance, the type could not be inferred, so we had to use <code>as</code> to let the compiler understand the right type.</p> <p>However, using this keyword in this case creates a false confidence feeling. <strong>The purpose of <code>as</code> is neither to protect nor to ensure typing</strong>, its main use being informing us about the hypothetical type of the object. For instance, in our previous example, I could fetch a book without knowing that the API has changed its definition of the type <code>Book</code>. Indeed, we could imagine that the type of the <code>id</code> attribute has changed from <code>number</code> to <code>string</code>, when we write:</p> <div class="gatsby-highlight" data-language="typescript"><pre class="language-typescript"><code class="language-typescript"><span class="token keyword">const</span> book <span class="token operator">=</span> <span class="token function">fetch</span><span class="token punctuation">(</span><span class="token string">"baseUrl/get-book?id=5"</span><span class="token punctuation">)</span> <span class="token keyword">as</span> Book<span class="token punctuation">;</span> <span class="token keyword">const</span> nextBookId <span class="token operator">=</span> book<span class="token punctuation">.</span>id <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">;</span></code></pre></div> <p>TS thinks that <code>book.id</code> is still a number, which could result in errors.</p> <p>So, once we figured out the purpose of <code>as</code>, we started by implementing what would later turn out to be a quick fix. We kept the <code>as</code> keyword, but added <strong>checks</strong> right after the problematic occurrences to make sure that the object’s attributes were not <code>undefined</code>:</p> <div class="gatsby-highlight" data-language="typescript"><pre class="language-typescript"><code class="language-typescript"><span class="token keyword">const</span> book <span class="token operator">=</span> <span class="token function">fetch</span><span class="token punctuation">(</span><span class="token string">"baseUrl/get-book?id=5"</span><span class="token punctuation">)</span> <span class="token keyword">as</span> Book<span class="token punctuation">;</span> <span class="token comment">// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>book<span class="token punctuation">.</span>author <span class="token operator">!==</span> <span class="token keyword">undefined</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">const</span> authorLastName <span class="token operator">=</span> book<span class="token punctuation">.</span>author<span class="token punctuation">.</span><span class="token function">split</span><span class="token punctuation">(</span><span class="token string">" "</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></code></pre></div> <p>This could prevent the problems cited above. However, it was not sustainable in the long-term, because we had to remember to perform the check every time we used an object whose type had been defined with <code>as</code>. Moreover, an "ignore" annotation was needed since the type was clear for our code linter <code>eslint</code>, which considered that the type defined by <code>as</code> was correct.</p> <p>Yet, some of the bugs persisted, until one day, we discovered <strong>type guards</strong> through a workshop. Little did we know, at the time, that they could solve most of our problems!</p> <h3>But first, what are type guards?</h3> <p>A type guard is a function that allows you to narrow the type of an object to a more specific one by performing certain checks. Hence, after calling the function, TypeScript understands the new type of the object.</p> <p><strong>typeof</strong> and <strong>instanceof</strong> are two built-in examples of such type guards:</p> <ul> <li> <p><strong>typeof</strong> only returns one of the following: "string", "number", "bigint", "boolean", "symbol", "undefined", "object", "function":</p> <div class="gatsby-highlight" data-language="typescript"><pre class="language-typescript"><code class="language-typescript"><span class="token keyword">typeof</span> <span class="token number">90</span> <span class="token operator">===</span> ‘<span class="token builtin">number</span>’ <span class="token keyword">typeof</span> <span class="token string">"abc"</span> <span class="token operator">===</span> ‘<span class="token builtin">string</span>’</code></pre></div> </li> <li> <p><strong>instanceof</strong> checks whether an object is of a specific type, for example:</p> <p><code>new Date() instanceof Date === true</code></p> <p>However, <strong>instanceof</strong> does not work with TypeScript interfaces, which are what we usually use.</p> </li> </ul> <p>Since neither operator could solve our issues, we decided to build our own brand new type guards. Let us consider for instance an <code>element</code> object whose type is <code>unknown</code> being actually of type <code>Book</code>. A working function allowing us to verify this assertion would be:</p> <div class="gatsby-highlight" data-language="typescript"><pre class="language-typescript"><code class="language-typescript"><span class="token keyword">const</span> isBook <span class="token operator">=</span> <span class="token punctuation">(</span>element<span class="token operator">:</span> <span class="token builtin">unknown</span><span class="token punctuation">)</span><span class="token operator">:</span> element <span class="token keyword">is</span> Book <span class="token operator">=></span> Object<span class="token punctuation">.</span>prototype<span class="token punctuation">.</span><span class="token function">hasOwnProperty</span><span class="token punctuation">.</span><span class="token function">call</span><span class="token punctuation">(</span>element<span class="token punctuation">,</span> <span class="token string">"author"</span><span class="token punctuation">)</span> <span class="token operator">&amp;&amp;</span> Object<span class="token punctuation">.</span>prototype<span class="token punctuation">.</span><span class="token function">hasOwnProperty</span><span class="token punctuation">.</span><span class="token function">call</span><span class="token punctuation">(</span>element<span class="token punctuation">,</span> <span class="token string">"id"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre></div> <p>A custom type guard takes an element in input and returns a <em><a href="https://www.typescriptlang.org/docs/handbook/2/narrowing.html#using-type-predicates">type predicate</a></em>, which in our example would be <code>element is Book</code>. Now, whenever we handle an element that could be a <code>Book</code>, we can use this function to check its type. Also, once the variable verifies the checks, TypeScript interprets its type as a <code>Book</code> allowing us to access its attributes:</p> <div class="gatsby-highlight" data-language="typescript"><pre class="language-typescript"><code class="language-typescript"><span class="token keyword">const</span> gift<span class="token operator">:</span> <span class="token builtin">unknown</span> <span class="token operator">=</span> <span class="token function">getGift</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">isBook</span><span class="token punctuation">(</span>gift<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token comment">/* * In this if-block the TypeScript compiler actually * resolved the unknown type to the type Book */</span> <span class="token function">read</span><span class="token punctuation">(</span>gift<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// read needs an argument of type Book</span> <span class="token keyword">return</span> gift<span class="token punctuation">.</span>author<span class="token punctuation">;</span> <span class="token punctuation">}</span></code></pre></div> <p>The other advantage of type guards is that we can unit test them, preventing problems:</p> <p>Here is an example with <a href="https://jestjs.io/">jest</a> (the JS testing framework we use):</p> <div class="gatsby-highlight" data-language="typescript"><pre class="language-typescript"><code class="language-typescript"><span class="token function">it</span><span class="token punctuation">(</span><span class="token string">"should return true if the element is a book"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span><span class="token punctuation">{</span> <span class="token keyword">const</span> element1 <span class="token operator">=</span> <span class="token punctuation">{</span> author<span class="token operator">:</span> <span class="token string">"alpha"</span><span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token function">expect</span><span class="token punctuation">(</span> <span class="token function">isBook</span><span class="token punctuation">(</span>element1<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toBe</span><span class="token punctuation">(</span><span class="token boolean">false</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> element2 <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">"id"</span><span class="token operator">:</span> <span class="token number">2000</span><span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token function">expect</span><span class="token punctuation">(</span> <span class="token function">isBook</span><span class="token punctuation">(</span>element2<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toBe</span><span class="token punctuation">(</span><span class="token boolean">false</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> element3 <span class="token operator">=</span> <span class="token punctuation">{</span> author<span class="token operator">:</span> <span class="token string">"alpha"</span><span class="token punctuation">;</span> <span class="token string">"id"</span><span class="token operator">:</span> <span class="token number">2000</span><span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token function">expect</span><span class="token punctuation">(</span> <span class="token function">isBook</span><span class="token punctuation">(</span>element3<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toBe</span><span class="token punctuation">(</span><span class="token boolean">true</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre></div> <p>Still, as we can see above, our type guard is incomplete. An object such as <code>{ author: 1904; "id": new Date()}</code> would be wrongly considered to be of type <code>Book</code>.</p> <p>To avoid this mistake, the function should also check the attributes’ types. Hence, We can modify it as follows:</p> <div class="gatsby-highlight" data-language="typescript"><pre class="language-typescript"><code class="language-typescript"><span class="token keyword">export</span> <span class="token keyword">const</span> isBook <span class="token operator">=</span> <span class="token punctuation">(</span>element<span class="token operator">:</span> <span class="token builtin">unknown</span><span class="token punctuation">)</span><span class="token operator">:</span> element <span class="token keyword">is</span> Book <span class="token operator">=></span> <span class="token comment">/* * We can not use "in" because element * has type "unknown" */</span> Object<span class="token punctuation">.</span>prototype<span class="token punctuation">.</span><span class="token function">hasOwnProperty</span><span class="token punctuation">.</span><span class="token function">call</span><span class="token punctuation">(</span>element<span class="token punctuation">,</span> <span class="token string">"author"</span><span class="token punctuation">)</span> <span class="token operator">&amp;&amp;</span> <span class="token keyword">typeof</span> element<span class="token punctuation">.</span>author <span class="token operator">===</span> <span class="token string">"string"</span> <span class="token operator">&amp;&amp;</span> Object<span class="token punctuation">.</span>prototype<span class="token punctuation">.</span><span class="token function">hasOwnProperty</span><span class="token punctuation">.</span><span class="token function">call</span><span class="token punctuation">(</span>element<span class="token punctuation">,</span> <span class="token string">"id"</span><span class="token punctuation">)</span> <span class="token operator">&amp;&amp;</span> <span class="token keyword">typeof</span> element<span class="token punctuation">.</span>id <span class="token operator">===</span> <span class="token string">"number"</span><span class="token punctuation">;</span></code></pre></div> <p>However, this fix will not be effective since the element type would still be unknown, preventing us from accessing its attributes directly:</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/b2aa5336e7d88b0bb080e1819762ab01/50383/object-is-unknown-error.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 25.945945945945947%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAFCAIAAADKYVtkAAAACXBIWXMAAAsTAAALEwEAmpwYAAABEUlEQVQY0z2MXU+CUABA70VuwPLbnKRrvQcWBgFXhgiaF5WYiiEw8yOrB8rW/39Ka7Wdp3O2A7IKZiQMBBWKGiWqULiDogpE7WDA0fxIUYNN/Zdj+gdJbfrGoK911vTAYA3JhurHRXdR91cFsgC9GLrbjBNC04fWNKP2wdVh+rc4VYxq26xoVpWE/Pw1N1rmR8uy98RPNvlhws9eyv4203vMukluvGQwQUKrKMioiY8LTsO1e6lk4ux4xYZfTPCJpmkheKvH+1qU1uP38yhtJHtuskP+jiVzrmtfGlbDNk70DqCkTqUrn1kSsiP08IHIGpBnxpnVBiTnBtxoUfKCC29IYRdiQqkOUCxwa9GyDlv4G2z9OIi85JsFAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="object is unknown error" title="object is unknown error" src="/static/b2aa5336e7d88b0bb080e1819762ab01/50383/object-is-unknown-error.png" srcset="/static/b2aa5336e7d88b0bb080e1819762ab01/1d79a/object-is-unknown-error.png 185w, /static/b2aa5336e7d88b0bb080e1819762ab01/1efb2/object-is-unknown-error.png 370w, /static/b2aa5336e7d88b0bb080e1819762ab01/50383/object-is-unknown-error.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>No worries, the deliverance could result from 2 options:</p> <ul> <li>either we split the function into 2 type guards, verifying the attributes' existence first, then their types each time we handle types</li> <li> <p>or, writing up a generic function <code>hasAttributes</code> that could check any attribute’s actual presence, permitting us to scale up the types’ checking. An example of such usage could be:</p> <div class="gatsby-highlight" data-language="typescript"><pre class="language-typescript"><code class="language-typescript"><span class="token keyword">export</span> <span class="token keyword">const</span> isBook <span class="token operator">=</span> <span class="token punctuation">(</span>element<span class="token operator">:</span> <span class="token builtin">unknown</span><span class="token punctuation">)</span><span class="token operator">:</span> element <span class="token keyword">is</span> Book <span class="token operator">=></span> <span class="token function">hasAttributes</span><span class="token punctuation">(</span>element<span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token string">"author"</span><span class="token punctuation">,</span> <span class="token string">"id"</span><span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token operator">&amp;&amp;</span> <span class="token keyword">typeof</span> element<span class="token punctuation">.</span>author <span class="token operator">===</span> <span class="token string">"string"</span> <span class="token operator">&amp;&amp;</span> <span class="token keyword">typeof</span> element<span class="token punctuation">.</span>id <span class="token operator">===</span> <span class="token string">"number"</span><span class="token punctuation">;</span></code></pre></div> </li> </ul> <p>Finally, we can access the attributes because we have checked their existence first.</p> <h3>Wait a second, how can we define <code>hasAttributes</code>?</h3> <p>The function takes an object of unknown type and returns a type predicate depending on the attributes given as an input to the function. In other terms, it needs to be a "<a href="https://www.typescriptlang.org/docs/handbook/2/generics.html">generic</a>", defined for instance in the following way:</p> <div class="gatsby-highlight" data-language="typescript"><pre class="language-typescript"><code class="language-typescript"><span class="token keyword">export</span> <span class="token keyword">const</span> hasAttributes <span class="token operator">=</span> <span class="token operator">&lt;</span><span class="token constant">T</span> <span class="token keyword">extends</span> <span class="token class-name"><span class="token builtin">string</span></span><span class="token operator">></span><span class="token punctuation">(</span> element<span class="token operator">:</span> <span class="token builtin">unknown</span><span class="token punctuation">,</span> attributes<span class="token operator">:</span> <span class="token constant">T</span><span class="token punctuation">[</span><span class="token punctuation">]</span> <span class="token punctuation">)</span><span class="token operator">:</span> element <span class="token keyword">is</span> Record<span class="token operator">&lt;</span><span class="token constant">T</span><span class="token punctuation">,</span> <span class="token builtin">unknown</span><span class="token operator">></span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>element <span class="token operator">===</span> <span class="token keyword">undefined</span> <span class="token operator">||</span> element <span class="token operator">===</span> <span class="token keyword">null</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token boolean">false</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token keyword">return</span> attributes<span class="token punctuation">.</span><span class="token function">every</span><span class="token punctuation">(</span><span class="token punctuation">(</span>attribute<span class="token punctuation">)</span> <span class="token operator">=></span> Object<span class="token punctuation">.</span>prototype<span class="token punctuation">.</span><span class="token function">hasOwnProperty</span><span class="token punctuation">.</span><span class="token function">call</span><span class="token punctuation">(</span>element<span class="token punctuation">,</span> attribute<span class="token punctuation">)</span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre></div> <p><br> The type <code>Record&#x3C;T, unknown></code> means that the <code>element</code> object in input does possess the wanted attributes but their values are of type <code>unknown</code>.</p> <h3>This is all great, but can we automate it a bit?</h3> <p>It is true that many type guards libraries already exist, like <a href="https://github.com/vedantroy/typecheck.macro">typecheck.macro</a>, but we have not had much luck using them. The main pain point with this library for example was that it makes our unit tests fail, we could not make it work with jest for the present time. Also, some other libraries require us to change all our type declarations to fit theirs in order to use them.</p> <p>However, since we now know how important type guards are and how much they can ease the developer experience, their automation could be feasible but is at the time being still a challenge.</p> <h3>Conclusion</h3> <p>As we have seen, we have nearly stopped using the <code>as</code> assertion to state objects’ types, since it is very error-prone and rarely necessary. In contrast, type guards have proven to be amazing tools, making our code <strong>cleaner</strong>, <strong>more stable</strong>, and <strong>more robust</strong>. In most cases, they can replace all of your <code>as</code> usages. As a matter of fact, my colleagues and I were not able to come up with any good occurrence which could not be replaced with a type guard, however hard we tried. Indeed, anytime you are faced with a type incoherence (a wrongly typed library, external data etc.) you can resolve it using a type guard.</p> <p>If you still are not convinced about their enormous potential, try them for yourself. Write your first type guard and get ready to see your developer experience improved!</p><![CDATA[Why You Should Encrypt Your Disk, and How To Do It on Ubuntu with a Dual Boot]]>/2022/01/disk-encryption-with-tutorial//2022/01/disk-encryption-with-tutorial/Mon, 10 Jan 2022 00:00:00 GMT<h2>Why use disk encryption?</h2> <p>It goes without saying that your computer can contain sensitive information about you or the organization you are working for, such as:</p> <ul> <li>personal pictures</li> <li>password or credit card numbers saved in your browser (and so on, access to your drive, important online accounts such as administrative ones, bank...)</li> <li>emails</li> <li>administrative documents (come one, we all store scans of our id or our social security number somewhere...)</li> <li>code or projects</li> <li>financial or operational spreadsheets</li> </ul> <h3>Why is it dangerous?</h3> <h4>Stolen identity</h4> <p>Your social security number, your id, or other administrative documents can be used to open <strong>bank accounts or credits</strong> in your name!</p> <h4>Fraudulent banking transaction</h4> <p>Stolen card numbers are very often sold to groups that will use them to perform online payments on the internet. The point is you can not realize that your number has been stolen until they use it. <strong>Then it is too late</strong>.</p> <h4>Blackmailing</h4> <p>You or your organization can be the victim of blackmailing. You for your personal pictures, for example and your organization for sensitive information, blueprints or source code.</p> <h4>Cyber-attack</h4> <p>If the stolen machine belongs to a company and contains employee credentials (VPN access for example), they can be used as an entry point on your company network, for a bigger cyber-attack such as <strong>ransomware</strong>.</p> <h4>Source code leak</h4> <p>If the source code of one of your projects leaks, it makes it easier for hackers to find <strong>security breaches</strong> in your product or application. It would also make it easier for rival companies to copy your product.</p> <h3>How is it technically possible and how to prevent it?</h3> <h4>Your password does not protect your files and data</h4> <p>Even without your password, someone with access to your hard disk can mount it on another operating system (for example by using an Ubuntu live USB key, a very common thing) and access some of the files it contains! It is an <strong>easy hacking practice</strong>. Nowadays, <strong>shady organizations buy second-hand laptops or hard drives to collect sensitive information about their former owner and sell them</strong>.</p> <p>In this video, the French Youtuber Micode shows us how he managed to collect sensitive data and files from second-hand hard disks he bought online:</p> <p style="position: relative; width: 100%; padding-top: 56.25%;"> <iframe style="position: absolute; top:0; left: 0;" width="100%" height="100%" src="https://www.youtube.com/embed/vt8PyQ2PGxI" title="YouTube video player" frameborder="0" allow="autoplay; clipboard-write; encrypted-media; picture-in-picture" allowfullscreen></iframe> </p> Also in January 2021, Kaspersky published on their blog an edifying study on data exposed in second-hand devices: **90% of the second-hand devices contain sensitive data!** <blockquote> <p>"Kaspersky’s Global Research and Analysis Team (GReAT) has examined security in secondhand devices. [...] An overwhelming majority of the devices the researchers examined contained at least some traces of data — mostly personal but some corporate — and more than 16% of the devices gave the researchers access outright. Another 74% gave up the goods when the researchers applied file-carving methods. A bare 11% had been wiped properly."</p> <p><a href="https://www.kaspersky.com/blog/data-on-used-devices/38610/">Sarah Pike on Kaspersky's blog, Jan.29 2021</a></p> </blockquote> <h4>Encryption to the rescue!</h4> <ul> <li><strong>Encryption is a way to prevent that</strong>, as an encrypted file can not be understood without the key to decode it.</li> <li> <p>On Ubuntu, you can encrypt your home folder after installation to protect the files inside. This is a very convenient practice, however it comes with major trade-offs:</p> <ul> <li>Encryption is handled at the <strong>application level</strong>. As a consequence, it can be <strong>slow</strong>, especially when you need to copy or download lots of files (when installing software or project dependencies for example).</li> <li>Your computer also contains files outside your home directory that can also provide <strong>information on you</strong>, such as the installed library or softwares for example. Also, some programs keep information in temporary files or log files. And any information can be temporarily stored in you swap space.</li> </ul> </li> <li>Full disk encryption leverages those two trade-offs as it provides <strong>faster kernel-level encryption</strong>, and encrypts <strong>your whole Ubuntu installation</strong> (except the boot partition).</li> </ul> <hr> <h2>Tutorial part: How to set up full disk encryption on Ubuntu</h2> <p>This article will not cover dual boot set-up, and assume that you either already have a fully-functional dual boot on your device, or your device is ready for installing Ubuntu on dual boot, or you don't need it.</p> <p>If you want to set up a dual boot, or enable Windows 10 disk encryption alongside Linux disk encryption, I recommend you switch to <a href="https://www.mikekasberg.com/blog/2020/04/08/dual-boot-ubuntu-and-windows-with-encryption.html">this longer, more complete tutorial</a> by Mike Kasberg.</p> <h3>(optional) Back up your data if you want to restore it after installation</h3> <p>Disclaimer: you will still have to reinstall your apps, but they should restore their current settings after reinstallation.</p> <ol> <li> <p>(optional): remove all node modules, venv, composer dependencies, ...</p> <p>First, you might not want to backup folders like node_modules or other project libraries as they can be very long to copy due to the very large amount of file they contain. Moreover, it is easy to reinstall them after your reinstallation.</p> <p>You can remove them manually or use the following command lines to remove all the folders with a given name in your home directory. Feel free to adapt them to the project you have.</p> <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash"><span class="token comment">#in your home directory</span> <span class="token builtin class-name">cd</span> ~ <span class="token comment">#delete all your node modules</span> <span class="token function">find</span> <span class="token builtin class-name">.</span> -name <span class="token string">'node_modules'</span> -type d -prune -exec <span class="token function">rm</span> -rf <span class="token string">'{}'</span> <span class="token comment">#delete all your python venv (supposing you named them venv)</span> <span class="token function">find</span> <span class="token builtin class-name">.</span> -name <span class="token string">'venv'</span> -type d -prune -exec <span class="token function">rm</span> -rf <span class="token string">'{}'</span></code></pre></div> </li> <li> <p>copy your home to an external drive. You can copy your entire home, or select only some folders you want to keep. Here are a few files and folders you might want to keep if you want to keep your apps settings and preferences:</p> <ul> <li>.ssh</li> <li>.config</li> <li>Documents</li> <li>Downloads</li> <li>...</li> </ul> <p>This list is not exhaustive and might depend on the apps you use</p> </li> </ol> <h3>Reinstall Linux with full disk encryption</h3> <p>For this you will need a live CD/usb key of Ubuntu.</p> <p>This tutorial has been tested on Ubuntu 20.04 LTS.</p> <h4>If you want to keep your dual boot</h4> <p>Unfortunately, the "Install Ubuntu alongside windows 10" option in the Ubuntu installer does not provide full disk encryption. We will need to set it up ourselves. If you don't need a dual boot you can skip to <a href="#no-dual-boot-tutorial">the next section</a></p> <ol> <li><strong>Select "Try Ubuntu" as you will need to use a few tools to prepare your disk before the installation</strong></li> </ol> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/2236f42aabbfc091f47525253d942abf/07f3a/ubuntu-installer-welcome.jpg" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 66.48648648648648%; position: relative; bottom: 0; left: 0; background-image: url('data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAANABQDASIAAhEBAxEB/8QAFgABAQEAAAAAAAAAAAAAAAAABAAB/8QAFgEBAQEAAAAAAAAAAAAAAAAAAQAC/9oADAMBAAIQAxAAAAFcLZVEjX//xAAbEAACAgMBAAAAAAAAAAAAAAABAgAUAwQSEf/aAAgBAQABBQKtj4OsnJwIJcfy48O05n//xAAUEQEAAAAAAAAAAAAAAAAAAAAQ/9oACAEDAQE/AT//xAAUEQEAAAAAAAAAAAAAAAAAAAAQ/9oACAECAQE/AT//xAAdEAACAgEFAAAAAAAAAAAAAAAAAQIhERIxMnGh/9oACAEBAAY/Ak8FRvsVemNMTjE2R//EABwQAQACAwADAAAAAAAAAAAAAAEAESExQRBRcf/aAAgBAQABPyGwCtrrChV6ym+THtABoffEI2z/AP/aAAwDAQACAAMAAAAQkx//xAAWEQEBAQAAAAAAAAAAAAAAAAARAAH/2gAIAQMBAT8QNsS//8QAFREBAQAAAAAAAAAAAAAAAAAAABH/2gAIAQIBAT8QR//EAB0QAQEAAgEFAAAAAAAAAAAAAAERACHRQVFh4fH/2gAIAQEAAT8QGxkrad9UxncCPZrI6ppOpXzhYgANcs+O84aBhNDzn//Z'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Ubuntu installer welcome" title="Ubuntu installer welcome" src="/static/2236f42aabbfc091f47525253d942abf/07f3a/ubuntu-installer-welcome.jpg" srcset="/static/2236f42aabbfc091f47525253d942abf/d7fe6/ubuntu-installer-welcome.jpg 185w, /static/2236f42aabbfc091f47525253d942abf/f4308/ubuntu-installer-welcome.jpg 370w, /static/2236f42aabbfc091f47525253d942abf/07f3a/ubuntu-installer-welcome.jpg 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <ol start="2"> <li> <p><strong>Define 2 disk partitions:</strong> one that will be encrypted and contain logical volumes for swap and root and one for boot</p> <ol> <li>Open GParted</li> </ol> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 628px; " > <a class="gatsby-resp-image-link" href="/static/93057a559752fae1945002daf077eb3c/b4e32/gparted-icon.jpg" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 46.48648648648649%; position: relative; bottom: 0; left: 0; background-image: url('data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAJABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAQBBf/EABUBAQEAAAAAAAAAAAAAAAAAAAAD/9oADAMBAAIQAxAAAAHj0SFNB//EABkQAQACAwAAAAAAAAAAAAAAAAIAAQMRIP/aAAgBAQABBQIqzFlWuP/EABQRAQAAAAAAAAAAAAAAAAAAABD/2gAIAQMBAT8BP//EABURAQEAAAAAAAAAAAAAAAAAABAR/9oACAECAQE/AYf/xAAZEAACAwEAAAAAAAAAAAAAAAABEQACIGH/2gAIAQEABj8CYgVz3P8A/8QAGRABAAIDAAAAAAAAAAAAAAAAAQARICEx/9oACAEBAAE/IUbaZtEs6DH/2gAMAwEAAgADAAAAEHjv/8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAwEBPxA//8QAFREBAQAAAAAAAAAAAAAAAAAAABH/2gAIAQIBAT8QhH//xAAcEAEAAwACAwAAAAAAAAAAAAABABEhEDFBYYH/2gAIAQEAAT8Qa24q6HPsMSGDG+sgZPHB1P/Z'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Gparted icon" title="Gparted icon" src="/static/93057a559752fae1945002daf077eb3c/b4e32/gparted-icon.jpg" srcset="/static/93057a559752fae1945002daf077eb3c/d7fe6/gparted-icon.jpg 185w, /static/93057a559752fae1945002daf077eb3c/f4308/gparted-icon.jpg 370w, /static/93057a559752fae1945002daf077eb3c/b4e32/gparted-icon.jpg 628w" sizes="(max-width: 628px) 100vw, 628px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <ol start="2"> <li>Delete your previous Ubuntu partition if you have one (To do so select the partitions and click on the red button in the top bar. Pay attention not to delete any windows partition). Your disk should look like something like (some unused space):</li> </ol> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/1bd197a18461bf505346b552db2735bf/07f3a/gparted-1.jpg" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 58.37837837837838%; position: relative; bottom: 0; left: 0; background-image: url('data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAMABQDASIAAhEBAxEB/8QAFwAAAwEAAAAAAAAAAAAAAAAAAAMEBf/EABYBAQEBAAAAAAAAAAAAAAAAAAABAv/aAAwDAQACEAMQAAABdfmWNMEif//EABkQAAMBAQEAAAAAAAAAAAAAAAACAxMEAf/aAAgBAQABBQLll40slMVOazLHdjdj/8QAFREBAQAAAAAAAAAAAAAAAAAAARD/2gAIAQMBAT8BZ//EABURAQEAAAAAAAAAAAAAAAAAAAEQ/9oACAECAQE/AUn/xAAaEAEBAAMBAQAAAAAAAAAAAAABABExMpEC/9oACAEBAAY/Aj6TNweXJ4QGLRaL/8QAGhABAAIDAQAAAAAAAAAAAAAAAQAxEBFxgf/aAAgBAQABPyGxZbCLlfjBBJabb7jWb//aAAwDAQACAAMAAAAQq9//xAAWEQEBAQAAAAAAAAAAAAAAAAABABH/2gAIAQMBAT8QTra3/8QAFhEBAQEAAAAAAAAAAAAAAAAAABEh/9oACAECAQE/ENEf/8QAIBABAAIABQUAAAAAAAAAAAAAAQARITFBUdFhobHB8f/aAAgBAQABPxDHggRY0pqX3hgKmz+Yi0ymMwlhvN1nyHmMfQ8z/9k='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Gparted before partition creation.jpg" title="Gparted before partition creation.jpg" src="/static/1bd197a18461bf505346b552db2735bf/07f3a/gparted-1.jpg" srcset="/static/1bd197a18461bf505346b552db2735bf/d7fe6/gparted-1.jpg 185w, /static/1bd197a18461bf505346b552db2735bf/f4308/gparted-1.jpg 370w, /static/1bd197a18461bf505346b552db2735bf/07f3a/gparted-1.jpg 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <ol start="3"> <li> <p>Then create 2 new partitions:</p> <ol> <li> <p>A partition for boot:</p> <ul> <li>size = 500Mio</li> <li>file-system = ext4</li> <li>label = boot</li> </ul> </li> <li> <p>A partition that will be containing swap volume and your Ubuntu installation:</p> <ul> <li>size = &#x3C;the size you want for your Ubuntu, including swap></li> <li>file-system = unformated</li> <li>label = root</li> </ul> </li> </ol> <p>At this point, your disk should look like something like this (2 new partitions instead of the unused space)</p> </li> </ol> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/8bee1f9fae4d51379540af9eaa846fa8/50383/gparted-2.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 63.78378378378379%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsSAAALEgHS3X78AAACFklEQVQ4y62T3UsUYRTG548t3ZX1gygi+5AuxA+ELropvwi8KgLJUshVlBAvukjR3DVtTWdzd+d73tmZeWd2dX+9rlRaUFAeeHiec+Y5h3PgHS17rYOuTIauzqxChmxHJ5nrneSyXfTkurl98xYPHwxwv/8u9/r727Xe7nP05HJt7uvppVfhzo0+NLNyjHdcJvBMXM9FCI84Esg0Jm02iBNJJOM2x4obaUKaJDQbDRKVO7ZNIATC8/B9H21qaQbdeIMT5rHCJcwgz9becwoHLzGUtqNlnGgVt+mxuVMgdm2aUUDsOezvbHP8pdTWSeATOxba2uY6hltEyAMsr4BX38d0d7H8PUyvSM3exnAKBEnAka4jhWpUVwSq2bcMnFoFGQhCNTR0HXWyLSjpVSwrQK/YStcQYRMjidFdF0dI6skpZ+EEdcqmxYnSZcOkqVjIhMNKlaNqjVTlWmlulg+zLzicf4W+tMDnt6+pvltmNz9PcXGe2toKJ8UtWp8+crSyyO7CHI2dTcqreeob73Her1NaXOCrypvbG2rDsQGcZ4/wp8cIp0YIJ0eIp0fxJ8cQM49xx4cQTwdVfRhvchRP+eoTQzjjwwTK604oqPpZb/3JIJqMkvY5tH6i9V2ftvgRrV/ABb7wXYsj+Zv/T/1/gxoYn5vVWleBSwOvIjQZy0uF/96wWjl7gxaGaSKlJE1TEvVr/Su+AdxntfSSsVRsAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Gparted after partition creation" title="Gparted after partition creation" src="/static/8bee1f9fae4d51379540af9eaa846fa8/50383/gparted-2.png" srcset="/static/8bee1f9fae4d51379540af9eaa846fa8/1d79a/gparted-2.png 185w, /static/8bee1f9fae4d51379540af9eaa846fa8/1efb2/gparted-2.png 370w, /static/8bee1f9fae4d51379540af9eaa846fa8/50383/gparted-2.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>Note the name of your root partition (here /dev/nvme0n1p6) and boot partition (here /dev/nvme0n1p5). Apply your changes and then you can close GParted!</p> </li> <li> <p><strong>Encrypt your main partition and split it in two logical volumes for swap and root.</strong></p> <p>For this, open a terminal and use these commands:</p> <ol> <li>get superuser rights: <code>sudo su</code></li> <li>encrypt your boot+swap partition: <code>cryptsetup luksFormat &#x3C;your_root_partition></code> (e.g. in my case: <code>cryptsetup luksFormat /dev/nvme0n1p6</code> ). Type <code>YES</code> to validate and choose a passphrase (that you will need to unlock your disk at boot time)</li> <li>open the encrypted partition: <code>cryptsetup luksOpen &#x3C;your_root_partition> ubuntu</code> (in my case: <code>cryptsetup luksOpen /dev/nvme0n1p6 ubuntu</code>)</li> <li>create a physical volume to access this partition: <code>pvcreate /dev/mapper/ubuntu</code></li> <li>create a volume group to contain your future logical volumes (root and swap) inside this physical volume: <code>vgcreate vgubuntu /dev/mapper/ubuntu</code></li> <li> <p>Inside this volume group, create 2 logical volumes:</p> <ul> <li>one for swap (here I use a size of 4G, feel free to adapt your swap size according to your hardware and needs. Usually swap should be around half your ram): <code>lvcreate -n swap -L 4G vgubuntu</code></li> <li>one for ubuntu (root), that will use all the remaining space: <code>lvcreate -n root -l +100%FREE vgubuntu</code></li> </ul> </li> <li> <p><strong>Optional</strong>: If you want to format your partitions using CLI, you can format them right now. Otherwise, this can be done using the Ubuntu installer:</p> <ul> <li><code>mkswap /dev/mapper/vgubuntu-swap</code></li> <li><code>mkfs -t ext4 /dev/mapper/vgubuntu-root</code></li> </ul> </li> </ol> <p>At this point, your terminal should look like this:</p> </li> </ol> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 737px; " > <a class="gatsby-resp-image-link" href="/static/fba5116c5e2533d8dc1a90e3c99d2293/d125e/terminal-lunks-encryption-and-lofical-volumes.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 104.86486486486486%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAVCAYAAABG1c6oAAAACXBIWXMAAAsSAAALEgHS3X78AAADg0lEQVQ4y3VUa3faRhTkpGma2s3DdpvGiQ2YNwiQ0BOweAsECAyOT/uh//9/TO9cIac+Of1wz12tpNmZ2dnNJckOvt+HbTtwPRembcJ2ba1VvMLh8Yh4HWMllewSnbNdB/NogU2yxXg6gRv4GIVD/DP1kQt7JoJOE06zAq/cRVj3cV8T4M8NePkugqIlvaNjv2BKdWFdN3XsfDXQ+VRD66oM47KEsfyfO5gzHMw5Ns0QSWOKY2cpFeHBEAbVMbb1CeJyqH2RD7Aq3SOR8b41x/zWx/Cqi9mNh8FFG7O7AXKJMcei2Id1VkX3bQnmr2WYZ2X0fqvq2DqrnObSrt/I2DqvwHlfh/2uDvdDQ99N70TypjVVwMEfXdz/aZ26SLto6XN43cMsL0w+mQg/p2PvYzNd5D/Vfl1IARelAUbXNuYCOiv40gPE1RCRzK+kzwsE6+pPvfOqFllq/SJs34oKmWv/XEwBn7wt9u0FvjlrPPZiBZrcuMokKg0xlTEZsSjN+9BE//cOgktDi7JpyzPgUUAIdDSXeOhG2BkzJGID62itsG1OMP7qYHHX1z7Leyo9uGqrJbSG7DsZYFwd4e9gh21jgofOQtkeBPybvdbORQi6l3fZgpHsJhWYJ+kvJC8roQLu2nNhNRPpGwVe18fKUNkKyE7SsBQ76DMBuXHPSTivvAT8y0+wIUNhkLHgM1nRJ0qjbwMpbh6lU7Incq2z8ksPVyeGqeRImXBMxnspxogAZDT6YquX01tPN46L/cCQwX7srZTRk7vBk0imZ2S6aYylL+V5KvNrfc80rGsjlZ7l8YWHe2FFr2L5iCCPdqweku34S09y2Jfgp9nciK+UTIZkSyuyfHbe3H2PDQG5KneYm0Hg5OQjN4B5JCjtWVXuJTaW2uC8qykzgjZf3WJS9JCLykMFY0SYx2xjDiKVcwczEpaBBn4p37IIqJsk/upYNohHM6oNkWu/KeoRYh75EyUtTrKYgMmNowuua6Fmj34ShIEOLtsKzLF/YWBRkdtmLbL4EaXE8lN6jof6c092j95QlvE6L72A1k95GNJJJCNDya1X+VQyvUoEUE+ISKWHmkPpHHMBRoWShoyO3D7ZrUSp3zelmAKOJU+UyFxRJpnRcOYwO/zex5bKYkx8mcvG7vvGjzl05Abhj7xJXLlJ9OPTjcKr6fmq+p/iKWHRmtGtg38BXf2OG4r0uVwAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="terminal commands luks and logical volumes" title="terminal commands luks and logical volumes" src="/static/fba5116c5e2533d8dc1a90e3c99d2293/d125e/terminal-lunks-encryption-and-lofical-volumes.png" srcset="/static/fba5116c5e2533d8dc1a90e3c99d2293/1d79a/terminal-lunks-encryption-and-lofical-volumes.png 185w, /static/fba5116c5e2533d8dc1a90e3c99d2293/1efb2/terminal-lunks-encryption-and-lofical-volumes.png 370w, /static/fba5116c5e2533d8dc1a90e3c99d2293/d125e/terminal-lunks-encryption-and-lofical-volumes.png 737w" sizes="(max-width: 737px) 100vw, 737px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <ol start="4"> <li> <p><strong>Now that all your partitions and logical volumes are ready, you can start your Ubuntu installation (finally)!</strong></p> <ol> <li>Open the installer, and follow the installation process until you get to this screen:</li> </ol> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/48785adb912ac51f247245f698ef9c51/50383/ubuntu-chose-installation-type.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 75.13513513513513%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAPCAYAAADkmO9VAAAACXBIWXMAAAsSAAALEgHS3X78AAAB+UlEQVQ4y4WT7ZKaMBSGubHujtZ2d9rZH3sJveOd6XQGRBAdFTUkIQqCC/ZtzsGwuP368XLOCclzPgjew90Yj+MJHkcTPIw6++lu1Ovjh3s8f33C85cn9ofvPt936vwxvk3G8OJZhGk4hR8EiOKIJZVCbnLovJM5HKxMH5Pyga9zzfvFbgdP7ASEsNoLKCmx3+9xPBYoyxJlUaIoChsfWeQ7cXx8i8/nMw7m0AGzLMPW0rdbpy1WqxXW6zUnECJjS/tIkhMLXnPrp9MJucrhGW1gbIuJ7yOKIsxmM8RxjOVyic1mA601A6RUbJXd60Qxgeq6RlVXMNoCc/s42v6XFpjM5wjDkKFJknCFVO3OVk+VkO38ros0TXtoD9TSDlTagXJrAsYYO7uC26CY5lNVFR96L5qz82+AWSqwWCy4TaqAQNyebZPAbla07uZFSZwcMHfA1+aV50AaZh9W8D/dAIctDTP/TTcgV2F1BSoLdJX1h07/Pshf9WrdGt3DK1D9BhwC3kPpI9FfQrFWurt/7u8hoMwkb+R5ladeHFf2jp3rXnSYgHwT7HsCkaU4t6IE3sH+p+2lRdM0aNqrrE9rVVlBbSR0qlh1VfMH5P1t82bbFpefF07m+d8DLKIF5uEcySzpFHY28iP4Lz6Cl4A1D+I/7nNr0x9T/ALxrEzOofzR4QAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Ubuntu chose installation type" title="Ubuntu chose installation type" src="/static/48785adb912ac51f247245f698ef9c51/50383/ubuntu-chose-installation-type.png" srcset="/static/48785adb912ac51f247245f698ef9c51/1d79a/ubuntu-chose-installation-type.png 185w, /static/48785adb912ac51f247245f698ef9c51/1efb2/ubuntu-chose-installation-type.png 370w, /static/48785adb912ac51f247245f698ef9c51/50383/ubuntu-chose-installation-type.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>Pick "Something else" and Continue</p> <ol start="2"> <li> <p>You should see all your disk volumes. Here we need to tell Ubuntu which volume to use for which use.</p> <ul> <li>Select /dev/mapper/vgubuntu-root, click Edit and pick the following settings (check format if you didn't do it with CLI): <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/6c89fd5b8b89f2fc5c6b09fcadb3ba5f/50383/ubuntu-installer-root-partition.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 71.35135135135134%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAAAsSAAALEgHS3X78AAACbklEQVQ4y4WSWU/bUBCF/Tsr+pQE+oJY2kf6GypVavvQN4IKVKiLqrKpEi2ITUWlbMFZTcgGtnO9Xi844XTuDa4CqlTLn86MJzl3xh5lLJvFk9HcPcZy2b/kMhlMTUzg6dSkjIdrw4xmM5gYH4diWw6YaUts5sKxPCRx/x5xmEgePn9IFMZQOuYlNOMEF8YZcYoL8wyX7FxqSl1SkJrmD2vCQ7ebUNTGT2wcLmH7eBl7hVXsnC5j/3wNansP563d/9Mc6FljC02zDMVyfHSdGDwGoh5wi4GGd0R90mQQixpNBrplTo/vXQkVlCv1BNWNL6htraG+vQ52tIOwdIigsA9+ukvsgZePEKoHiKrHCMu/EVGdUx5QLPKwMqjzlgZF/7UN7d1LaEtvUXv/BtriaxhfZ+Gu5OGvzMITbH6GtZwHW1+AtT4PtjqHYGMJ/MdH8O8fwDc/gX9bgF84gGLSV9auDDQMBhbEcOIevOQWvaFR5GsgAgpCUn47GDfl5k5DmpnWxoLn0Op0TVodE9zzYHW7aDUuh2ig3Wyi02qRNmAaOnzXgevY8r+eK1bPIHXpozALJhl1yUQQRRHm8nk8HhnBaC6HSVrq5zMzeDY9jRzlmZFHmH/1AnGSgHMuCYJAImLFJdeUtFCv11Gr1VCpVFAulVAsFqGqqlRRq1arsG0bvu9LPJrKoU5FrPieL7sKw1AiTmGMIYojWIaFdr2Fq+trtNtt6LqBXq8n6wJhJBDNiANkh8JQGInOUu10OjAMg94rw018g36/L43iOJaHCzPRkcjTRlL+aXhNHRm6Lk11UpFrmiYPShGdpb8f5g/jvOn9dDW37wAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Root partition settings" title="Root partition settings" src="/static/6c89fd5b8b89f2fc5c6b09fcadb3ba5f/50383/ubuntu-installer-root-partition.png" srcset="/static/6c89fd5b8b89f2fc5c6b09fcadb3ba5f/1d79a/ubuntu-installer-root-partition.png 185w, /static/6c89fd5b8b89f2fc5c6b09fcadb3ba5f/1efb2/ubuntu-installer-root-partition.png 370w, /static/6c89fd5b8b89f2fc5c6b09fcadb3ba5f/50383/ubuntu-installer-root-partition.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></li> <li>Select /dev/mapper/vgubuntu-swap, click Edit and pick the following settings: <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/1f4b9be5b518ffa12df934ef6e401b6b/50383/ubuntu-installer-swap-partition.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 71.89189189189189%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAAAsSAAALEgHS3X78AAAChklEQVQ4y5WTy08TURTG57/kIRjQalwgeyQujCvF+Eg0xo0YBIOIbkxcUFAKXehKBanQlmqF6bxn7p339MHnuXeoEIwLJ/Prd8+597xmpsro0DDOD58wSowMDmFkKOfcwAAuXyzgSqEg133/XwwOojA+DsVlHJppQzcdmI4Hy+WIkjZCIvof0g7CKIESuAa4ugfeqoOpNdIaQqOBQG/A1+o5rfrJ+l+0qghdHYpRKaOyMovd9SXUNl+hurGE3Q+LtF6Gtb1KFP+oubUCa0to8djOfUKNz+/gNb9BYdyH4QYIs54k7gBB0oEXpuBxG0HaBY9ybQOIsiPQLe2kC9ANCpGadnpQbNPAXmUH1e8V7O5so9mow7UMxIGPdhIjSyK4hwZ8z4GhqbB0Dcyx4TkWTFrbpg7L0OBQDGceFOtjET+e3MDB/F3oiw9gEOzNY/gv74HN3wJfmAF7+xR8+SHY60dgpJyUL9yWe5IXd8Cf3YT/qQjFcxz8bOzDs230Om10s1R2hq4Y5OQSfvS6uXFEY8ch2nEkyY41DkMoURwjTlIwzuG4LoIgRJq1Ua3VsLq2hlJpA5ubZZTLZayXSnhPvsrXLwgpLqAEQRjR50KPxWPSVhhjME0TNnXoeR5cShrRgedzc7gwNoarExO4NjWF69PTmJycxCX6eGfvz8iEMSHO9lWgiB/f54QvN5IkAaduRZEsy6Sd0ASnLx4EVDwvHFJXgoB8QpU8IKefUKhlWTKAUzE5hWXKCcSeKCYmEU2IRKIBsScTxset9+knFEEtVcVh8wC/Gk3U9+pQD1Wo5NM0TQanaYrTDQmUsw6ZNCalN+3Rf1tvamjtq6Q6uJc/GtF5/+xZfgPhQOgWSU320AAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Swap partition settings.png" title="Swap partition settings.png" src="/static/1f4b9be5b518ffa12df934ef6e401b6b/50383/ubuntu-installer-swap-partition.png" srcset="/static/1f4b9be5b518ffa12df934ef6e401b6b/1d79a/ubuntu-installer-swap-partition.png 185w, /static/1f4b9be5b518ffa12df934ef6e401b6b/1efb2/ubuntu-installer-swap-partition.png 370w, /static/1f4b9be5b518ffa12df934ef6e401b6b/50383/ubuntu-installer-swap-partition.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></li> <li>Select the boot partition you created in step 1-c (/dev/nvme0n1p5 in my case), and pick the following settings: <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/1cd1fcb983936cb481d1ca634e9fc1e8/50383/ubuntu-installer-boot-partition.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 71.35135135135134%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAAAsSAAALEgHS3X78AAAC5klEQVQ4y22T229UVRSHzx+nD6iJSS2XVrQ29Ik3X/BPIBD0QVB8aKQxBIIBW0CSCpSE0rRCAiSCdqZlembmdGbO/X6bS+dz7UMn9sGTfFlr39Ze+7fW0WanTzB7/CSfC7OHzEwfF05UnJqa5syXX7EwN1/5k3lFdVb2KmY+m2b+9BdonuviOBaObeG5Dr7vUhYZxYQ8k3H+fi4/Mi/keU6Sl+8p+iRpjmbZPvq+TVNodZwKw/Qwej77pkL5Hu3DcbV2ONezHFK7SWrppOYeeeigvanprDzcZHX9NbdXt7j3aIvf115wf32Du+LflbUHT17wcO0Zy39scu/xn9x/vMXK2ksePHrK32s3qD+9xfb6EsY/z9HiMCQMXBiPGA5KoWDQz8myiCjwyJKIIotJI1+enDIoMw6G/crPlRwj6I+hEJvK0zXHdtip12m12mRpymg0wvd8CZjRL/uiXUGe5XQ7HSzLFq1tSSAkTRLiKCKNI7k0ZjRQl4iGrh+w22xRa+iEWYH6knLAmP8+5XtxQs8L6DqineuTDkZEkpFC7T+QPcVgiOa/2sC8dRlzZRH/zo8kd37A+vUKxs3LdK5/h3HtPDvXvqf5yyV2Fs/T+PkCe0sXiW9fIVm+SvKbsPwT0c1viV8/Ew2tLml7l2Cvhl1/Q7/XYvXGEqc/+oD5T49x9tQU5xbm+HpuhoWpT5j7+EMWvzmL926bSK8Tyjll3dpfqFiaH0Q4ookrNkhSInl2QyRoNNvUdhvSHiZd0fmd3qRpdGjtd3i7LUFE12J4QC7PnJAVUpQ4iitxFbkUQuE6TlUMVaREtAtVJ4SBFCNgOBwSiA18v1pXxVEksXSCjLVUsirLUjq/qDpfWVX5WDZ4UoDWThPDMGi323Q63ar6jqP+KL/ak0iwSJJRl6SyVgWcBJsENE0TW7WHZKYODuQ54/G4yk5drg6rIJNEjvK/AVUGKuDEKnRdp9vt0uv1KqsyO3puwr/x9+AzB7FpIgAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Boot partition settings.png" title="Boot partition settings.png" src="/static/1cd1fcb983936cb481d1ca634e9fc1e8/50383/ubuntu-installer-boot-partition.png" srcset="/static/1cd1fcb983936cb481d1ca634e9fc1e8/1d79a/ubuntu-installer-boot-partition.png 185w, /static/1cd1fcb983936cb481d1ca634e9fc1e8/1efb2/ubuntu-installer-boot-partition.png 370w, /static/1cd1fcb983936cb481d1ca634e9fc1e8/50383/ubuntu-installer-boot-partition.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></li> </ul> </li> <li>Click install now to start the installation.</li> </ol> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/2399ea75af28f85660609d3bfb4fb0da/50383/ubuntu-installer-install-now.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 16.216216216216214%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAADCAYAAACTWi8uAAAACXBIWXMAAAsSAAALEgHS3X78AAAAaElEQVQI14WPvQ6AIAyEef+XBFyUCSj/LqdHYsJgdPjStL1ermqzFlprGGPgnMO+H/Deo/eO1torz451pZQClXOGiCClNIfnGJ9mhDe11llpQmKMs1erkKI/qOMHNAkhQO4gNON3DHUBhIrnSG0e/vQAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Install now" title="Install now" src="/static/2399ea75af28f85660609d3bfb4fb0da/50383/ubuntu-installer-install-now.png" srcset="/static/2399ea75af28f85660609d3bfb4fb0da/1d79a/ubuntu-installer-install-now.png 185w, /static/2399ea75af28f85660609d3bfb4fb0da/1efb2/ubuntu-installer-install-now.png 370w, /static/2399ea75af28f85660609d3bfb4fb0da/50383/ubuntu-installer-install-now.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>Your partition changes should look like this (if you formated them in the installer):</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/acdfe469d4041f78a3bb1e3d31bb4506/50383/ubuntu-installer-partition-change-verify.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 34.5945945945946%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAHCAYAAAAIy204AAAACXBIWXMAAAsSAAALEgHS3X78AAAA9klEQVQoz42Qy26DQAxF5x9bIFEbddvS/i8PAWV4J6EgMbzE5nbsaCoUdZHF0bU9tuFaWE/PONg2nHssCy/HIz5dF+77ByvxdjrB1m8Hx+E5g2XZ+Hp1IJI4RlGWKIoSeZ4jyzLWW3zL0zRlvjWUSykhtWZaTc59MoOIokgvK7hAWurlRF3XaJoGVVVBKYVhGFiVGji+x7yJMAxBBEGAJElwuV65oe97ZhxHLMvyMIIW+b4Pz/MQa/uUk92u69C2LWbdNM/zw7Blcx++jUzZ5rqujGlcdov/i//+kGxt28bDVDC6/+p+aF+j09Cdp2nC5XzWjn7wC1NE7kBQLyKKAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Partition change confirmation.png" title="Partition change confirmation.png" src="/static/acdfe469d4041f78a3bb1e3d31bb4506/50383/ubuntu-installer-partition-change-verify.png" srcset="/static/acdfe469d4041f78a3bb1e3d31bb4506/1d79a/ubuntu-installer-partition-change-verify.png 185w, /static/acdfe469d4041f78a3bb1e3d31bb4506/1efb2/ubuntu-installer-partition-change-verify.png 370w, /static/acdfe469d4041f78a3bb1e3d31bb4506/50383/ubuntu-installer-partition-change-verify.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>You can finish the installation process.</p> <p>⚠️ If you want to restore your home folder, you need to create a user with the same name and password as your original one!</p> <p>⚠️ <strong>Do not restart your device at the end of installation!</strong> At this point, Ubuntu is installed but it does not know that your disk is encrypted. Thus it is not able to boot yet. We will configure it in the next part.</p> </li> <li> <p><strong>Setup <code>/etc/crypttab</code>, for Ubuntu to know that then encryption is encrypted and how to unlock.</strong></p> <p>To do so, click "keep trying" at the end of the installation, and open the terminal. We will mount your new Ubuntu partition and use chroot on it to edit this /etc/crypttab and apply the changes.</p> <ol> <li>Get superuser rights: <code>sudo su</code></li> <li> <p>Get your encrypted partition UUID ( ⚠️ not the PARTUUID), it will be needed to set up the crypttab:</p> <p><code>blkid &#x3C;your_encrypted_partition></code> (e.g in my case: <code>blkid /dev/nvme0n1p6</code>).</p> <p>Keep it for later.</p> </li> <li>Mount ubuntu: <code>mount /dev/mapper/vgubuntu-root /target</code></li> <li>Mount your boot partition: <code>mount &#x3C;your_boot_partition> /target/boot</code> (i.e. in my case <code>mount /dev/nvme0n1p5 /target/boot</code></li> <li> <p>use chroot on the /target and mount everything:</p> <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash"><span class="token keyword">for</span> <span class="token for-or-select variable">n</span> <span class="token keyword">in</span> proc sys dev etc/resolv.conf<span class="token punctuation">;</span> <span class="token keyword">do</span> <span class="token function">mount</span> --rbind /<span class="token variable">$n</span> /target/<span class="token variable">$n</span><span class="token punctuation">;</span> <span class="token keyword">done</span> <span class="token function">chroot</span> /target` <span class="token function">mount</span> -a</code></pre></div> </li> <li> <p>Now, you can edit /etc/crypttab on the new installation (I'll do it with nano for the example):</p> <p><code>nano /etc/crypttab</code></p> <p>Copy/paste this inside the new crypttab (don't forget to replace your encrypted partition UUID!):</p> <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">ubuntu <span class="token assign-left variable">UUID</span><span class="token operator">=</span><span class="token operator">&lt;</span>your_encrypted_partition_id<span class="token operator">></span> none luks,discard</code></pre></div> </li> <li> <p>Last but not least: apply the modifications of the crypttab:</p> <p><code>update-initramfs -k all -c</code></p> </li> </ol> </li> </ol> <p>🥳<strong>You're done! Congrats, you can exit your live CD and restart your computer!</strong></p> <p>If everything worked as expected, you will be asked your passphrase to unlock your disk when you start Ubuntu.</p> <h4><span id="no-dual-boot-tutorial">If you don't need a dual boot</span></h4> <p>If you don't need a dual boot on your computer, you can also erase your entire disk and install Ubuntu on it. This process is much easier. On the Installation Type screen:</p> <ul> <li>select "Erase disk and install Ubuntu".</li> <li>Before continuing click on the "Advanced Features" button and select "Use LVM" and "Encrypt the new Ubuntu installation for security".</li> </ul> <p>You can go along with the Ubuntu installer then.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 377px; " > <a class="gatsby-resp-image-link" href="/static/7380aada7e20031f3b0c46205102b08b/6146e/ubuntu-installer-auto-encrypt-single-boot.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 61.08108108108108%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAIAAADtbgqsAAAACXBIWXMAAAsSAAALEgHS3X78AAACBElEQVQoz31PW4/SYBDtX/IFWhZYXlZ3YbcguglZ190oT/4BE/+dMb6pCdIW6EfvpZSWSwt8paXcPGDUp/Wk+Tpz5szMGeau0TjjuFKxeF4o4Cvm81eXV6/q9etKuVbla3y1xvPXlcrl8xfn+TwEpUKByxeaF2efGjdMpVzOZjL5XC7HchzLIq7yfPN98/Hh7bvHh9vXt3jv39zXX9bYbBaCHMs+y7AfLriPNyVGEARJlFqtliRJ8gmdTkdot4UTJEnEK4rC3yohpE+I1CPffwrMYrGIV7Ft2UtKD4fDfr8//BcQ7HY7BBGljPi1rQvqOJy4Q9dxHNM0B/ZgCLjuYDCwbRuk67qWZZum4ThDSul6vU7TzWw2Y4yurnwjrS8/NE01DIMQWVU1VVFgsA8ofVVVVFXtdrtwi9zz/PF47HkehjIhnYduIH1uy3251+vhYOigJkRBAEZRFPwkUez1ZCITZziEHxjUdJ3xPT9N0yiJXPh24RyOHd/3R94IDMaPsAUi33eHx3S5pNBvt9swDJnJZAJOFESp05WPu2WYNy1L13VV02DcMg0cYugG4iAI03QdRVGSJNPplMEBcRxHR9Df+BdRusSiI/OnRKPVapWccGzG5s1mg2T9BNADh/P5fBXH6yTZbjezIEDL0TYuwnjUFk/A9z0iy31FCU4j4AIXgcfmXy/4Q2tV9g/9AAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Advanced features to enable disk encryption" title="Advanced features to enable disk encryption" src="/static/7380aada7e20031f3b0c46205102b08b/6146e/ubuntu-installer-auto-encrypt-single-boot.png" srcset="/static/7380aada7e20031f3b0c46205102b08b/1d79a/ubuntu-installer-auto-encrypt-single-boot.png 185w, /static/7380aada7e20031f3b0c46205102b08b/1efb2/ubuntu-installer-auto-encrypt-single-boot.png 370w, /static/7380aada7e20031f3b0c46205102b08b/6146e/ubuntu-installer-auto-encrypt-single-boot.png 377w" sizes="(max-width: 377px) 100vw, 377px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h3>(Optional) Restore your data</h3> <p>Log in on your new installation and create a new user (superuser). Log out from the user you want to restore and log in on the new superuser. Then you can copy/paste your home or the files you want to restore to their proper emplacement on the home of the user you are currently not logged in.</p> <p>⚠️ As said earlier, for the restoration to work properly some files will need both of your users (former and new ones) to have the same name and the same password! You can change your password later.</p> <p>Once it is done, you can log back on the other user and delete the other superuser if you do not need it.</p> <h2>Conclusion</h2> <p>By following this tutorial, you learned about various fields:</p> <p><strong>About security</strong></p> <ul> <li><strong>Hardware security is an important part of cyber-security</strong> for organizations as well as for individuals. A device can be stolen, lost, or as you will probably throw or sell it at some point, and you want to make sure that the data inside it is not accessible if it happens.</li> <li><strong>Your password is not required</strong> to access some of the files inside your computer if someone has <strong>physical access</strong> to your hard disk. That can happen while you own it, or even after. <strong>To prevent it, you can encrypt your device.</strong></li> </ul> <p><strong>About device encryption</strong></p> <ul> <li>You can use <strong>Luks</strong> to encrypt a physical volume and then split it in logical volumes for your different needs (swap, root, optionally a dedicated home volume).</li> <li>Ubuntu can handle an encrypted physical volume if it knows it from <code>/etc/cryptab</code>, and then use its logical volumes as regular partitions.</li> <li>You can trick Ubuntu installer by unlocking an encrypted partition using <code>cryptsetup luksOpen</code>. Generaly speaking, <strong>cryptsetup</strong> is a tool to encrypt partitions and unlock them for your OS to use them (including the Ubuntu installer)</li> </ul> <p><strong>About Ubuntu</strong></p> <ul> <li><strong>You can copy a home folder</strong> from an Ubuntu installation and restore it on another one if you create a user with the same name and password. Unfortunately, libraries are not backed-up.</li> <li>GParted is a tool on Ubuntu to manage your physical partitions.</li> <li>Home folder encryption is not a good practice as the encryption is managed at the application level so it can often be slow. Moreover, you protect only your files. <strong>Full disk encryption</strong> is handled at the kernel level, hence it <strong>is much faster</strong>. It also ensures that absolutely no information can be extracted from your device, including your libraries, your swap space...</li> </ul> <p><strong>To go further</strong></p> <ul> <li> <p>If you want to know more about the process of encrypting-decoding data, and when to use file encryption:</p> <ul> <li>A video explaining <a href="https://www.youtube.com/watch?v=UPW1Hqvx6zo">How Does Full Disk Encryption Work?</a></li> <li>A video explaining <a href="https://www.youtube.com/watch?v=Re8IK--TMdk">How Does Individual File Encryption Work?</a></li> </ul> </li> <li>The article on Kaspersky blog about data collection in second-hand device: <a href="https://www.kaspersky.com/blog/data-on-used-devices/38610/">https://www.kaspersky.com/blog/data-on-used-devices/38610/</a></li> </ul> <p>Do not hesitate to browse <a href="https://blog.theodo.com/">Theodo's blog</a> for more interesting articles or visit <a href="https://www.theodo.fr/">Theodo's website</a> to learn more about us !</p><![CDATA[The React Testing Library Guide I Wish I Had]]>/2021/12/react-testing-library-guide//2021/12/react-testing-library-guide/Tue, 21 Dec 2021 00:00:00 GMT<p>I would like to share with you a great library I have found, which makes front-end tests fun to write!</p> <p>With this article, you will be able to write serenely your first tests with React Testing Library. To have a deeper understanding of the library you can use the <a href="https://testing-library.com/docs/react-testing-library/intro/">documentation</a>.</p> <h2>Is it really useful to write front-end tests?</h2> <p>The problem is that as your app grows you cannot be sure that everything still works properly each time you add some new code. Seems quite obvious, right? Then, why do so few people write front-end tests?</p> <p>When I asked some people about it, they answered: "we don't have enough time", "it's not really useful"...</p> <p>The first thing you should know is that, yes, at the beginning it seems that tests are not worth the time. But as your app grows it will be more and more difficult to implement them and your app will be more and more unstable. Then, <strong>you should implement tests from the beginning of a project</strong>: it is easier and it will be timesaver later.</p> <p>The second thing to know: <strong>tests are definitely useful</strong>. A quick anecdote to convince you: the first test I have ever written spotted a bug.</p> <p>Then it's time to learn how to test!</p> <h2>Why use React Testing Library?</h2> <p>Now that you are - I hope - convinced that you should test your front-end code, what library should you use?</p> <p>My first advice is: <strong>don't use snapshots</strong>, as this <a href="https://blog.theodo.com/2020/07/write-tests-for-humans/">article</a> explains. To sum it up: snapshots are difficult to understand for developers and then it doesn't really test the app the way users use it.</p> <p>When I implemented front-end tests on my project I chose <strong>React Testing Library</strong>. React Testing Library allows you to test by <strong>simulating human interactions with the UI</strong>, while being quite independent of the implementation.</p> <p>Indeed, the closer the test is to the actual app usage, the more relevant it is.</p> <p>As a bonus, using React Testing Library neatly can actually <strong>improve the accessibility</strong> of your app (<a href="https://juhanajauhiainen.com/posts/how-testing-library-will-improve-accessibility">See an article about accessiblity improvement</a> to understand why).</p> <h2>How to test with React Testing Library?</h2> <p>The principle of React Testing Library is to render a component and then simulate some interaction with it, and then finally compare the result with the expectation.</p> <h3>1. Render a component</h3> <p>The first thing you have to do is to get the component you want to test. It can be a whole page or isolated components. The best thing to do is to find the right balance between testing a page with a lot of components, which can be long because of nested components but is closer to the user experience, and testing components, which is faster but sometimes less relevant.</p> <p>Anyway, <strong>you should never test already tested components and behavior</strong>. For example, components from libraries are usually tested and you shouldn't test them again.</p> <p>The easiest way to get a component is to use directly the <strong>render method</strong>:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">import</span> <span class="token punctuation">{</span> render <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"@testing-library/react"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> MyComponent <span class="token keyword">from</span> <span class="token string">"../MyComponent"</span><span class="token punctuation">;</span> <span class="token function">test</span><span class="token punctuation">(</span><span class="token string">'My first test'</span><span class="token punctuation">,</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token function">render</span><span class="token punctuation">(</span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">MyComponent</span></span> <span class="token punctuation">/></span></span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></code></pre></div> <p>It is also possible to <strong>wrap the render function in a provider to mock some features</strong>, such as the Redux store or the internationalization:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">import</span> <span class="token operator">*</span> <span class="token keyword">as</span> React <span class="token keyword">from</span> <span class="token string">"react"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> RenderResult<span class="token punctuation">,</span> render <span class="token keyword">as</span> rtlRender<span class="token punctuation">,</span> RenderOptions <span class="token keyword">as</span> RtlRenderOptions<span class="token punctuation">,</span> <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"@testing-library/react"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> IntlProvider <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"react-intl"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> Provider <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"react-redux"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> flattenMessages <span class="token keyword">from</span> <span class="token string">"services/i18n/intl"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> frMessages <span class="token keyword">from</span> <span class="token string">"translations/fr.json"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> rootSaga <span class="token keyword">from</span> <span class="token string">"../redux/sagas-for-tests"</span><span class="token punctuation">;</span> <span class="token keyword">const</span> defaultMessages <span class="token operator">=</span> <span class="token function">flattenMessages</span><span class="token punctuation">(</span>frMessages<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> render <span class="token operator">=</span> <span class="token punctuation">(</span> ui<span class="token operator">:</span> React<span class="token punctuation">.</span>ReactElement<span class="token punctuation">,</span> <span class="token punctuation">{</span> initialState<span class="token punctuation">,</span> store <span class="token operator">=</span> <span class="token function">storeCreator</span><span class="token punctuation">(</span>initialState<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token comment">// custom function to create the store</span> messages <span class="token operator">=</span> defaultMessages<span class="token punctuation">,</span> <span class="token operator">...</span>renderOptions <span class="token punctuation">}</span><span class="token operator">:</span> RenderOptions <span class="token punctuation">)</span><span class="token operator">:</span> <span class="token parameter">RenderResult</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> Wrapper<span class="token operator">:</span> React<span class="token punctuation">.</span><span class="token function-variable function">ComponentType</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> children <span class="token punctuation">}</span></span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token punctuation">(</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">Provider</span></span> <span class="token attr-name">store</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>store<span class="token punctuation">}</span></span><span class="token punctuation">></span></span><span class="token plain-text"> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">IntlProvider</span></span> <span class="token attr-name">messages</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>messages<span class="token punctuation">}</span></span> <span class="token attr-name">locale</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>fr<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token plain-text"> </span><span class="token punctuation">{</span>children<span class="token punctuation">}</span><span class="token plain-text"> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span><span class="token class-name">IntlProvider</span></span><span class="token punctuation">></span></span><span class="token plain-text"> </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span><span class="token class-name">Provider</span></span><span class="token punctuation">></span></span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token function">rtlRender</span><span class="token punctuation">(</span>ui<span class="token punctuation">,</span> <span class="token punctuation">{</span> wrapper<span class="token operator">:</span> Wrapper<span class="token punctuation">,</span> <span class="token operator">...</span>renderOptions <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre></div> <p>With a custom render function like this you can render your component with a mocked state:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">import</span> <span class="token punctuation">{</span> mockState <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'__fixtures__/state'</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> render <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'utils/test-utils'</span><span class="token punctuation">;</span> <span class="token comment">//custom render</span> <span class="token keyword">import</span> MyComponent <span class="token keyword">from</span> <span class="token string">"../MyComponent"</span><span class="token punctuation">;</span> <span class="token function">test</span><span class="token punctuation">(</span><span class="token string">'My first test with a mock state'</span><span class="token punctuation">,</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token function">render</span><span class="token punctuation">(</span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">MyComponent</span></span> <span class="token punctuation">/></span></span><span class="token punctuation">,</span> <span class="token punctuation">{</span> initialState<span class="token operator">:</span> mockState <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></code></pre></div> <p>Now that you can display a component, let us play a bit with it.</p> <h3>2. Simulate user interactions</h3> <p>To interact with the elements of a component you will need a way to access them. You have several ways to do this. You can, for example, query elements by text, label, placeholder, alternative text, or role. Regardless of what attribute you use you have three main methods to get an element.</p> <p>Generally, I prefer to use <strong>getBy</strong>, which is the most simple one. But if you need to get the element asynchronously you can use <strong>findBy</strong>. It can be useful when you wait the results of an asynchronous method for example. If you want to check if an element is not in the DOM, I advise you to use <strong>queryBy</strong>, which returns "null" if the element was not found.</p> <p>You should know that all of these methods will throw if more than one element matches, though you can overcome this limitation by using getAllBy, findAllBy, or queryAllBy to get an array.</p> <p>Have a look at how you would query for a button reading “press me”:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">import</span> <span class="token punctuation">{</span> render <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"@testing-library/react"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> MyComponent <span class="token keyword">from</span> <span class="token string">"../MyComponent"</span><span class="token punctuation">;</span> <span class="token function">test</span><span class="token punctuation">(</span><span class="token string">'Get button test'</span><span class="token punctuation">,</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> <span class="token punctuation">{</span> getByRole <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">render</span><span class="token punctuation">(</span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">MyComponent</span></span> <span class="token punctuation">/></span></span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> pressMeButton <span class="token operator">=</span> <span class="token function">getByRole</span><span class="token punctuation">(</span><span class="token string">"button"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token string">"press me"</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></code></pre></div> <p>Once you have your button <strong>you can interact with it</strong>, by simulating a click on it:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">import</span> <span class="token punctuation">{</span> render <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"@testing-library/react"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> MyComponent <span class="token keyword">from</span> <span class="token string">"../MyComponent"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> user <span class="token keyword">from</span> <span class="token string">"@testing-library/user-event"</span><span class="token punctuation">;</span> <span class="token function">test</span><span class="token punctuation">(</span><span class="token string">'Click on button test'</span><span class="token punctuation">,</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> <span class="token punctuation">{</span> getByText <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">render</span><span class="token punctuation">(</span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">MyComponent</span></span> <span class="token punctuation">/></span></span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> pressMeButton <span class="token operator">=</span> <span class="token function">getByText</span><span class="token punctuation">(</span><span class="token string">"press me"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> user<span class="token punctuation">.</span><span class="token function">click</span><span class="token punctuation">(</span>pressMeButton<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></code></pre></div> <p>Another classical way to interact with an element is when the user types something in an input field. This interaction can also be easily simulated. With this code, for example, you would simulate a user typing "some random text" in an input field labeled "myInputField':</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">import</span> <span class="token punctuation">{</span> render <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"@testing-library/react"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> MyComponent <span class="token keyword">from</span> <span class="token string">"../MyComponent"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> user <span class="token keyword">from</span> <span class="token string">"@testing-library/user-event"</span><span class="token punctuation">;</span> <span class="token function">test</span><span class="token punctuation">(</span><span class="token string">'Fill input test'</span><span class="token punctuation">,</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> <span class="token punctuation">{</span> getByText <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">render</span><span class="token punctuation">(</span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">MyComponent</span></span> <span class="token punctuation">/></span></span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> myInputField <span class="token operator">=</span> <span class="token function">getByLabelText</span><span class="token punctuation">(</span><span class="token string">"myInputField"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> user<span class="token punctuation">.</span><span class="token function">type</span><span class="token punctuation">(</span>myInputField<span class="token punctuation">,</span> <span class="token string">"some random text"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></code></pre></div> <p>Sometimes you will try to get an element, but you will have an error message saying that it was not found. To help you debug that, React Testing Library has a wonderful method...</p> <h3>3. Debug</h3> <p>It is really easy to debug with React Testing Library. Indeed, it provides a method, <strong>debug</strong>, to display the entire DOM at a specific moment.</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">import</span> <span class="token punctuation">{</span> render <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"@testing-library/react"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> MyComponent <span class="token keyword">from</span> <span class="token string">"../MyComponent"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> user <span class="token keyword">from</span> <span class="token string">"@testing-library/user-event"</span><span class="token punctuation">;</span> <span class="token function">test</span><span class="token punctuation">(</span><span class="token string">'Debug fill input test'</span><span class="token punctuation">,</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> <span class="token punctuation">{</span> getByText<span class="token punctuation">,</span> debug <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">render</span><span class="token punctuation">(</span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">MyComponent</span></span> <span class="token punctuation">/></span></span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> myInputField <span class="token operator">=</span> <span class="token function">getByLabelText</span><span class="token punctuation">(</span><span class="token string">"myInputField"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">debug</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//displays the DOM with an empty input</span> user<span class="token punctuation">.</span><span class="token function">type</span><span class="token punctuation">(</span>myInputField<span class="token punctuation">,</span> <span class="token string">"some random text"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">debug</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//displays the DOM with the input filled with "some random text"</span> <span class="token punctuation">}</span></code></pre></div> <h3>4. Examine the result</h3> <p>Now that you are able to render a component and interact with it the way the user would do, it is time to <strong>check that everything works as expected</strong>.</p> <p>To do that you just have to use the expect method, for example, to check that an input has the expected text content:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">import</span> <span class="token punctuation">{</span> render <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"@testing-library/react"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> MyComponent <span class="token keyword">from</span> <span class="token string">"../MyComponent"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> user <span class="token keyword">from</span> <span class="token string">"@testing-library/user-event"</span><span class="token punctuation">;</span> <span class="token function">test</span><span class="token punctuation">(</span><span class="token string">'Debug fill input test'</span><span class="token punctuation">,</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> <span class="token punctuation">{</span> getByText <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">render</span><span class="token punctuation">(</span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">MyComponent</span></span> <span class="token punctuation">/></span></span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> myInputField <span class="token operator">=</span> <span class="token function">getByLabelText</span><span class="token punctuation">(</span><span class="token string">"myInputField"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> user<span class="token punctuation">.</span><span class="token function">type</span><span class="token punctuation">(</span>myInputField<span class="token punctuation">,</span> <span class="token string">"some random text"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">expect</span><span class="token punctuation">(</span>myInputField<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toHaveTextContent</span><span class="token punctuation">(</span><span class="token string">"some random text"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></code></pre></div> <h3>5. Mocks</h3> <p>Finally, you should know that <strong>you can mock the different API calls and hooks using Jest</strong> to fully test your app. Using mocks enables you to test highly complex functions. Moreover, it enables you to have control over the data.</p> <h2>Conclusion</h2> <p>Testing the front-end with React Testing Library <strong>improves the quality</strong> of your code itself, it enables you to <strong>test the user interactions</strong> and above all, it is <strong>quite intuitive</strong>. Now you no longer have any excuse to avoid testing your front-end!</p><![CDATA[Disrupting B2B payments & driving purchasing efficiencies for SMBs]]>/2021/12/disrupting-b2b-payments//2021/12/disrupting-b2b-payments/Mon, 13 Dec 2021 00:00:00 GMT<p>SMBs are in need of innovative and connected tools that solve these complex problems. Theodo and Codat partnered to produce a proof of concept application that showcases how fintechs and incumbent banks can help small businesses drive payment efficiencies by leveraging consented data.</p> <h3>What is wrong with B2B payments today?</h3> <p>The traditional ways in which businesses pay bills is over complicated and manual, leaving room for error which increases the cost and burden on SMBs.</p> <p>To help with growth, finance teams are increasingly expected to perform more strategic roles, yet many of the functions and processes they’re required to manage are archaic and in need of optimisation. Owing to these inefficient practices, the accounts payable team is often left struggling with mounting paperwork, the threat of late payments, data errors, and with no means of providing accurate management reporting. The following statistics outline the extent of this problem:</p> <ul> <li>Poor payment processes <a href="https://tradeshift.com/wp-content/uploads/ArdentPartners-State-of-ePayables-2019.pdf">cost up to $10</a> per invoices for SMBs.</li> <li>60% of businesses admit to <a href="https://www.cimaglobal.com/Documents/Thought_leadership_docs/white-paper-hub/2016-04-12-Invoice-Benchmark-Report.pdf">late payment fees</a> causing reputational damage and supplier friction.</li> <li>32% of businesses report <a href="https://www.cimaglobal.com/Documents/Thought_leadership_docs/white-paper-hub/2016-04-12-Invoice-Benchmark-Report.pdf">duplicate payments</a> for the same invoice.</li> <li>56% of businesses admit planning is a challenge because they <a href="https://www.cimaglobal.com/Documents/Thought_leadership_docs/white-paper-hub/2016-04-12-Invoice-Benchmark-Report.pdf">do not have visibility</a> of upcoming bills to pay.</li> </ul> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/89ed19be85a962944df3fc1544118bc5/50383/thumbnail.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 54.59459459459459%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAABeklEQVQoz5VSXU7CQBjkjCZexBv4wAF88Qj+xQcTnz2AiYmASFsKuLS0CC3ZLWzb/W0dLP4RfXCabrqzndnZ79tW/QGlVJ7nZVkWRSGVxIhp8Y6yKBs0pDG6kbTwVlWFMVkuer0n3x85zoBMiee63cfHgeNMxuOR7/ef+0N/OPS8TqdDGW1UrfobKmulENjB6p23RRwhQDYb7GEnblY453eM3lDKOEc4ZTSh/MhdX4dZrWUmrbbVn2Kp1O06u8jYC2Pt1/jkNe6vNge9zSlZ9wJ2eEmuugnSVHviTxTWIuKK8+NZ2A4C1CrRtTT6fpIeno/OHuK60vtii9OqLZpvURRpxjZlWRljtcJjtFpuD8OlkEBTgp2YUUoIiaIIep7nMyAKUVD8F4ThYrEwcDEmWS7DIIyjGNMv8Xw+D2czWIBdpakzGLiuI4RACV3PQ4uSJNHaoGfgCZkqpX/E/mwGLgJcKGNgcCteEGlKsiyDbxAE6SpFnN8L9i+8ATxkcKvTb8ZDAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Traditional payment process for SMBs" title="Traditional payment process for SMBs" src="/static/89ed19be85a962944df3fc1544118bc5/50383/thumbnail.png" srcset="/static/89ed19be85a962944df3fc1544118bc5/1d79a/thumbnail.png 185w, /static/89ed19be85a962944df3fc1544118bc5/1efb2/thumbnail.png 370w, /static/89ed19be85a962944df3fc1544118bc5/50383/thumbnail.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h3>What can you do to help small businesses?</h3> <p>SMBs are in need of innovative and connected tools that solve these complex problems. Theodo and Codat partnered to produce a proof of concept application that showcases how fintechs and incumbent banks can help small businesses drive payment efficiencies by leveraging consented data. The key objectives of this project were to produce a tool that would deliver the following benefits for SMBs:</p> <p>Visibility</p> <ul> <li>Allow businesses to see all of their outstanding bills, their value, and due date in one place to enable them to accurately forecast and schedule upcoming spend.</li> </ul> <p>Security</p> <ul> <li>Reduce invoice fraud and streamline payments validation by using open banking data and other tools to verify a supplier’s bank details before executing large payments.</li> </ul> <p>Automation</p> <ul> <li>Minimise friction between different business systems and make it easier to see what is outstanding and then make the payments in a single interface.</li> </ul> <p>Accuracy</p> <ul> <li>Drive better accuracy with payment amounts and the status of the bill.</li> </ul> <p>Speed</p> <ul> <li>Enable SMBs to focus on their growth by saving them hours of manual reconciliation</li> </ul> <h2>What did we build?</h2> <iframe src="https://player.vimeo.com/video/654568015" width="640" height="564" frameborder="0" allow="autoplay; fullscreen" allowfullscreen></iframe> <h3>Key Features:</h3> <h4>Authorisation</h4> <p>Rather than build directly to the accounting platforms, we chose to integrate with Codat. This meant we could rely on a single data model and did not have to write libraries and handlers for complexities such as token refresh and request throttling.</p> <p>We were able to save considerable engineering time by taking advantage of Codat’s <a href="https://bit.ly/3pFO2NQ">Link</a> product, an out-of-the-box and no-code solution that handles the authorisation process.</p> <h3>Retrieve list of bills</h3> <p>Once the connection is authorised by the user, Codat’s REST API makes it easy to retrieve a list of accounts payable invoices (bills) in a standardised format, meaning once we had built the integration for QuickBooks, it worked seamlessly for other platforms without requiring changes. For each bill we included information such as <a href="https://bit.ly/33g21md">line item detail and expense category</a> and even a jpg or pdf attachment of the invoice.</p> <h4>Mapping</h4> <p>It’s absolutely critical that the payment is recorded against the correct account in the business’s accounting system - Codat makes this easy by providing a list of accounts available in our API to query.</p> <h4>Reconcile payments</h4> <p>The final step of the puzzle involves reconciling the payment back to the business’s accounting platform. Codat’s API is bi-directional, which enables you to create a payment and reconcile this against outstanding bills. This ensures their accounting platform is always up to date and saves time with manual entry.</p> <h4>How did we achieve this?</h4> <p>Implementing agile practice, the Theodo team started with a proof of concept console application, to ensure the API calls were successful. The Codat Swagger made this process seamless, as it provided the developers with real-time visibility and a simple way of adopting these with Axios.</p> <p>The team moved on to building a dynamic frontend experience for users using google MUI react components and next.js, with particular focus on transaction consistency, to ensure all data is displayed correctly and latency does not cause any issues.</p> <p>“The tight feedback loop with the Codat team, combined with a focus on ease of operation, resulted in an ever-improving interface; for example, the team incorporated pagination and loading components to maintain user engagement. The API sandbox set-up meant the team could access the platform and create mock companies, integrating the user experience at every stage of the development process.”</p> <p>The cost of building an API integration can easily run into the thousands, depending on the number and complexity of the integrations. Codat’s API has a number of advantages that drastically reduce the integration complexity, therefore increasing the speed to market. The first is that only a single integration is needed, a key factor in reducing complexity. Secondly, the API is simple, meaning less work is needed for the system to communicate. The data relationships between systems are also standardised, ensuring clean data transfer. Finally, the precise documentation means that if any changes do need to be made, they can be actioned quickly and efficiently.</p> <p>The quality of the API offers clients a Portal with deep integrations to a vast number of external platforms, such as Xero and QuickBooks Online (QBO), as well as Desktop (QBD) and complex ERPs (including NetSuite and Intacct). These integrations allow for near real-time data syncs between systems, delivering instantaneous business value for a client.</p> <h3>How can Codat &#x26; Theodo help you build better SMB products?</h3> <p>Codat is the universal API for small business data. Codat provides two-way connectivity to over 30 different accounting, banking, and commerce systems used by small businesses, including Xero, Plaid, QuickBooks, Stripe, Oracle NetSuite, Shopify and Sage Intacct. Codat’s integrations are standardised to a single data model, allowing clients to build to Codat once rather than having to sink their own engineering resources into building and maintaining each complex integration themselves.</p> <p>Theodo partners with businesses of all sizes, helping to increase their market share by building bespoke digital products that users love. Lead-time is critical when reacting to market opportunity, thus pragmatic technical solutions are essential.</p> <p>Click <a href="https://www.codat.io/us/payables/?utm_campaign=Theodo-Bill-Pay&#x26;utm_medium=blog&#x26;utm_source=blog&#x26;utm_term=Bill-Pay&#x26;utm_content=Theodo-Bill-Pay">here</a> to learn more about how to build a B2B payments solution.</p> <p><em>SadiqD | Partner @ Theodo</em></p> <blockquote> <p>“The market is flooded with solutions that seem to solve every problem in the world. In reality, only a few do what they say on the tin. The more we’ve delivered solutions that centre around Codat, the more opportunities I see with it. For any business looking to help companies make the most from their financial data, Codat is truly a no-brainer. Our engineering team loves it, and the Codat team is so easy to partner with”.</p> </blockquote><![CDATA[Building Terravision(2021) using Deck.gl and Carto]]>/2021/12/terravision-2021//2021/12/terravision-2021/Tue, 07 Dec 2021 00:00:00 GMT<p>Adding a detailed and interactive map to a web app can be a time consuming process. Pairing deck.gl and carto with react can make it pain free. Using Carto to manage your data and Deck.gl to render it can make for a great developer and user experience. Resulting in a <a href="https://terravision2021.herokuapp.com/">web app</a> rich with detailed and engaging visualisations.</p> <blockquote> <p><strong>TLDR: skip to case study section if you're too cool for context...</strong></p> </blockquote> <h2>Intro</h2> <p>At the beginning of the Netflix show The Billion Dollar code, you’re introduced to two German developers, Juri Muller and Carsten Schlute. As the series progresses we learn more about Terravision, an application that the two developers and their team (ART+COM) develop that closely resembles the modern day map software we are all familiar with, Google Earth. The series takes place in the 90s and hence the algorithms and mapping software that we use today has progressed significantly from the first versions of Terravision and Google Earth.</p> <p>Nowadays, as a web developer, the inclusion of a map in your application can surpass the simple use cases provided by the google maps API. Whether it be needing to display a dynamic view of nearby taxis for your journey home or a heat map showing what people in your city are up to; maps in modern day applications are rich with information and data. Two tools that can be used to build rich map visualisations in web applications are Carto and Deck.gl.</p> <p>Simply put, Carto is a data warehouse that specialises in data for geospatial analysis with some magic under the hood.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/0c060cfa5fa87924e974e8301699a3b8/50383/carto-catalogue.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 47.56756756756757%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAIAAAA7N+mxAAAACXBIWXMAAAsSAAALEgHS3X78AAAB30lEQVQozy1QTWsUQRCd3yFeNSHgQVwxJFkVFfWgSQQPWTGOqyj4Czx48CQeYgT1qJBkleDVgHj0LB69KLo7M8km+zGzszM9Pd3VHzNt9c42j+JVV71+1eXsfotP3e5cehRce7K/6PqIBdevN4P6fcvrTZvWGv7lx/6zt+H1p3s3nm/P3fLefY5KI51PX+PZ1c6iGyDm1/0K59b92h3/TMOrNTyMp9csWbjnn737Z979feJm+83uRLz1JT5+9d9SM7jwIDjfDJbc4OLDfXSeWWnPrHQqzGJcbqPm5LI3t+ofu/J3Yyc0KP7xi778EL3+GL/aGVXYbE3JxnZUYVpqYU+02YpevI++/8yEko5Q2thTSEG0rrgpCyg0NUYYA+gwucMIZam0AptoTXLhcKHK0nAuUkKl1EoVSmnGgGSMcwkgOQgAlWUsTuhgJMdEG+wXKAYHpLK+RUHzHEBIKcuyBIBxkqZpRgjl9tgKA50ShtbYL+REjM6YHA1lfxCPxwkIHNWQjLa9g+5h3wu6B93eUT887A17/WGes+pfU3H1Z5yW5gw9qxpjfBCOwigexQnJcpw8ZxyJHUzaFaCKMOFQxnFPwCEMo5wx5EopHNtOKkRR6EJPgZe4BjRHju8lhP0H9rzm3zph35wAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Screenshot of Carto data catalogue" title="Screenshot of Carto data catalogue" src="/static/0c060cfa5fa87924e974e8301699a3b8/50383/carto-catalogue.png" srcset="/static/0c060cfa5fa87924e974e8301699a3b8/1d79a/carto-catalogue.png 185w, /static/0c060cfa5fa87924e974e8301699a3b8/1efb2/carto-catalogue.png 370w, /static/0c060cfa5fa87924e974e8301699a3b8/50383/carto-catalogue.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>If you’re building an application that requires demographic data detailing the populations of cities in Europe just search through the catalogue, get the URL of the data set and import it using their API. Carto is built with developers in mind and hence they have a great API that pairs really well with the second tool I’ll be talking about, deck gl. If you make a call to their SQL api to obtain the data you’ll see a result containing demographic data mapped to a set of long lat coordinates. This format (GEOJSON) is an industry standard that can be interpreted by most mapping software but can be tedious and cumbersome to generate manually so it’s great to have an easily accessible source.</p> <blockquote> <p><em>Note: For readers in the UK another great source of geospatial data is the <a href="https://geoportal.statistics.gov.uk/datasets/ons::local-authority-districts-may-2021-uk-bfe/about">ONS</a>.</em></p> </blockquote> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/0c7abdbafd897a977913c186b25eba8d/50383/ons-site.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 47.56756756756757%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAIAAAA7N+mxAAAACXBIWXMAAAsSAAALEgHS3X78AAAB2UlEQVQoz21Ry27TUBD1H7dlGdwgRKhoWAFCQkJddENRCRJVYdEgNZUaKbETvx/Xb1/X14kfiQlpA4nzYhKkiAWjszije47OzFyqXC5XKhWapkulEk0flkqPD/Yf/Rf7ewegOdzWk6fPnr84ppCBsixbr9er1SruZyTsu3bwL3wnsOzAAZj4Rz4CZTEvZvPldL6ifk9+zqa/5rMp2D3XZxmB5+QdOE5utkW2IwLvdkTTdNI0c10/IlGaJJQV5mE2uUsfvGRy1rBAJAmquAUQpqtWP3e+NKBRBV6RJE3XTU01BoPhcJhTb7+ikyvzpG6//+7vvWvxnLIzb9Si9rLWPv3WRrIGT1xXchzfcTzfD7AfUDEJtjPDJosbBoFi59RV45rRz+vsmwuW7SqKpHGcgnQrwHcQrqoGhZAxHt+DebFYepazM8P8noejfr/ZFl7Xbs/qnMDLmmYQEsX9GDbfjG077vj+YbVez4qFplkQuEuG82DXu2zwRx9vT694XlAw9ofbgg8ajUZUSKKiKDZftVyayIbAv04AD+Dkm5Z43ZIU3U7TNCQEYxzHkJzmeU4dfWCrn8RqTX51oTcZVZU1OJIi66qCAEBMZPUIGQyyJE3A0+tBR4AkcfwHmfbmWTJUhXwAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Screenshot of ONS Contour dataset" title="Screenshot of ONS Contour dataset" src="/static/0c7abdbafd897a977913c186b25eba8d/50383/ons-site.png" srcset="/static/0c7abdbafd897a977913c186b25eba8d/1d79a/ons-site.png 185w, /static/0c7abdbafd897a977913c186b25eba8d/1efb2/ons-site.png 370w, /static/0c7abdbafd897a977913c186b25eba8d/50383/ons-site.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>So now we have our maps and the data to enrich them. We need a way to combine them and display the result in our web app. Deck.gl is a software specialising in mapping and geospatial visualisation. It offers a react library designed to create high-performance web-GL based visualisations for large datasets. Hence it’s one of the easiest ways to integrate geospatial visualisations into your applications. The library is really intuitive and highly optimised out of the box. It also contains a bunch of different features that will fit a large number of use cases required by your application.</p> <h2>Case Study</h2> <p>Let’s talk through a real world example. Say you’re an international news company and you want to revamp the user experience of your website. You cover news stories all across the globe but you’ve gotten feedback from users saying finding new, interesting stories can be difficult and your team has proposed a new map interface to display all the current popular stories. As well as displaying the stories on the map you think it would be nice to supplement the articles with contextual information about the countries you’re covering (GDP, HDI, Life Expectancy, etc).</p> <p>Slight caveat. Unfortunately, Carto is not a free piece of software so for the example I will be using a dataset that I’ve downloaded from <a href="https://www.kaggle.com/datasets">Kaggle</a>. Hopefully talking through the full process that I have to follow to integrate a generic dataset into our application and how it differs from using Carto will allow me to highlight the convenience of using Carto in your workflow.</p> <h3>The Data</h3> <p>To begin with you’ll need to source the data for the contextual information. As noted above, a quick search on <a href="https://www.kaggle.com/datasets">Kaggle</a> you can find a list of demographic indicators for all countries in the world. Granted this is outdated but this is where something like Carto comes in handy as it provides up to date data in the required format.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/1f896023b171906bc24152c107ff190a/50383/kaggle-data-structure.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 84.32432432432432%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAARCAIAAABSJhvpAAAACXBIWXMAAAsSAAALEgHS3X78AAAA2klEQVQ4y52TyW4DIRBE5/+/1HLiMGy90N2My3IuyQlc4oLEU1FFcxBLOotQr/nUWrzX6X6t6Zgzau2tNREm4uk251yHZ0Q0cETSmwtd11zkj/e5lMvtfhciKrl12oNVxyOdiN3PhCuoxSr81hiO8lhH7oogezA2Zi5aGz0afQ2jDTgC5mamwxjLXDdgSHQwl04/jb6J0x6MqB7orrFkHX0D9ggGKILk24X5a1SYO2zB2zasaEwV5LYz2sY7Y0LlA2ckFdQtPDAuax/rF0ZJ65/pD/wqdy3kPz0BiLzwyCF/yqQAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Screenshot of Kaggle data schema" title="Screenshot of Kaggle data schema" src="/static/1f896023b171906bc24152c107ff190a/50383/kaggle-data-structure.png" srcset="/static/1f896023b171906bc24152c107ff190a/1d79a/kaggle-data-structure.png 185w, /static/1f896023b171906bc24152c107ff190a/1efb2/kaggle-data-structure.png 370w, /static/1f896023b171906bc24152c107ff190a/50383/kaggle-data-structure.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h2>Using Deck.gl with next.js and react</h2> <p>Now we have our data, the next task is to set up your project. Deck.gl works with react out of the box so you can use the <a href="https://nextjs.org/docs/api-reference/create-next-app">next project starter</a> to generate a react project and get going from there. Firstly, you’ll need to create a <a href="https://deck.gl/docs/get-started/using-with-react#the-deckgl-react-component">DeckGL component</a> for your page. This is where all of your geospatial visualisations will sit.</p> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"><span class="token operator">&lt;</span>DeckGL initialViewState<span class="token operator">=</span><span class="token punctuation">{</span><span class="token constant">INITIAL_VIEW_STATE</span><span class="token punctuation">}</span> controller<span class="token operator">=</span><span class="token punctuation">{</span><span class="token boolean">true</span><span class="token punctuation">}</span><span class="token operator">></span> <span class="token operator">&lt;</span>StaticMap mapboxApiAccessToken<span class="token operator">=</span><span class="token punctuation">{</span><span class="token constant">MAPBOX_KEY</span><span class="token punctuation">}</span> <span class="token operator">/</span><span class="token operator">></span> <span class="token operator">&lt;</span><span class="token operator">/</span>DeckGL<span class="token operator">></span></code></pre></div> <p>The above code snippet would display a simple map on the page where you render the DeckGL component. You will see there’s a child component <code>StaticMap</code>. This child handles the map element of our page. Deck.gl can easily integrate with <a href="https://www.mapbox.com/">mapbox</a> so all you need to do is create an account and use the public token to add a map to your Deck.gl component.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/bbdfcff25fb903b1b59d16ac4d7cac88/50383/basic-deckgl-map.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 29.18918918918919%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAGCAIAAABM9SnKAAAACXBIWXMAAAsSAAALEgHS3X78AAAA6klEQVQY002O60oDMRCF8+5dvKAvocX2R99Ft3QzSdhNMs1lWUtBsftHUPEkC+JwCJmP78CI2017t903jy/I6uH5enO42R7ud7pZtyvAdVt4TfMvMK+e9oIHQo62JHo9v71+f15+vubMJiJOB6cSm2DV4vzl/ZwFKdNJQkhp67x1fDqdPy6zJAWCV2njmfvBYi1mB1lh5WMQOWdrrfc+pTiOIzOHEACllMMwOOdiXVNKeKE5a0MsZoxR9H2v68ADQhMdNEGgorB4MSUIppoQwGEKIoLa4RqiBS0EBq4orZymacKH6sg60HDLL1EBLBI6Kg+qAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Screenshot of Deck.gl map on homepage" title="Screenshot of Deck.gl map on homepage" src="/static/bbdfcff25fb903b1b59d16ac4d7cac88/50383/basic-deckgl-map.png" srcset="/static/bbdfcff25fb903b1b59d16ac4d7cac88/1d79a/basic-deckgl-map.png 185w, /static/bbdfcff25fb903b1b59d16ac4d7cac88/1efb2/basic-deckgl-map.png 370w, /static/bbdfcff25fb903b1b59d16ac4d7cac88/50383/basic-deckgl-map.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>As you can see the map currently takes the whole of our page but we need to limit it to the hero section of the screen so we can see the articles as well. Adding the following prop to our Deck.gl component allows us to add some styling:</p> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">views<span class="token operator">=</span><span class="token punctuation">{</span><span class="token punctuation">[</span><span class="token keyword">new</span> <span class="token class-name">MapView</span><span class="token punctuation">(</span><span class="token punctuation">{</span> width<span class="token operator">:</span> <span class="token string">'100%'</span><span class="token punctuation">,</span> height<span class="token operator">:</span> <span class="token string">'44%'</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">}</span></code></pre></div> <p>The resulting page:</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/b948aeccff5fb199e99654d13217c556/50383/styled-deckgl-map.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 41.62162162162162%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAIAAAB2/0i6AAAACXBIWXMAAAsSAAALEgHS3X78AAABNklEQVQY02WN607CQBCF9+FBYmJiNPEFRIh/NCY+CGADvbDbvbe2lqYX2rIV1Gn5YRO+THZnzzkzi67ny5vn1Wi6GD0uxk/L2xfv4Z3ev2HoJ/MPOMfT5dVsBW5X08U5OZmt7l5dJHybbtecbKRvx9pPI/ndlq0pRac4UJq7XwFTzBXE5ngDSUHtJGRpLBEXaosJJj7jUumA+BQuKMtae1sMFhcSnkIqaECBADSQpIwjQojbQylVSjHGPM+zLMtxHOgDHQCiB2Ns2zYkQT8nkd9DfCKlTNM0SRIwIBqGYRzHUfQZRVGS7OAEETKcc631btcpKMuLfbkHTNv+XpDlHWVZ1nV96aJ9Vbc9xpjD4XA8Hn8GVHVjzL97Op2GLjrv7tcXVVXB8HB3nhdZnhX9503TwPDQ/QPyz6s210r44gAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Screenshot of styled Deck.gl map on homepage" title="Screenshot of styled Deck.gl map on homepage" src="/static/b948aeccff5fb199e99654d13217c556/50383/styled-deckgl-map.png" srcset="/static/b948aeccff5fb199e99654d13217c556/1d79a/styled-deckgl-map.png 185w, /static/b948aeccff5fb199e99654d13217c556/1efb2/styled-deckgl-map.png 370w, /static/b948aeccff5fb199e99654d13217c556/50383/styled-deckgl-map.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>So now we have our map displayed on the homepage of our website, the next task is to take the geojson data we downloaded from Kaggle and add it to the map. For this I’ve opted to store the layers in the state which is populated when the page mounts. Using <code>getStaticProps</code>, I read the json file from disk and pass it as a prop to the homepage. This prop is then used to dispatch a redux action that takes the json data as an argument and returns the layers.</p> <blockquote> <p><em>Note: The format of the geojson data is not fully inline with what deck.gl expects (one reason why using Carto would be nice here) and hence I have to do some transformation on data load. An alternative solution here would be to process the data once yourself and then have this loaded into the app when the page mounts.</em></p> </blockquote> <p>I handle the creation of the layers inside a redux saga. For the geojson data I create a new instance of the <a href="https://deck.gl/docs/api-reference/layers/polygon-layer">PolygonLayer</a> class.</p> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"><span class="token comment">// Create basic polygon layer from geojson data</span> <span class="token keyword">const</span> europeLayer <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">PolygonLayer</span><span class="token punctuation">(</span><span class="token punctuation">{</span> id<span class="token operator">:</span> <span class="token string">'real-poly-layer'</span><span class="token punctuation">,</span> data<span class="token operator">:</span> action<span class="token punctuation">.</span>payload<span class="token punctuation">.</span>geoJson<span class="token punctuation">,</span> <span class="token function-variable function">getPolygon</span><span class="token operator">:</span> <span class="token parameter">d</span> <span class="token operator">=></span> <span class="token punctuation">(</span>d <span class="token keyword">as</span> <span class="token punctuation">{</span> type<span class="token operator">:</span> string<span class="token punctuation">;</span> geometry<span class="token operator">:</span> <span class="token punctuation">{</span> type<span class="token operator">:</span> string<span class="token punctuation">;</span> coordinates<span class="token operator">:</span> Position<span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token punctuation">]</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">.</span>geometry<span class="token punctuation">.</span>coordinates<span class="token punctuation">,</span> getFillColor<span class="token operator">:</span> <span class="token punctuation">[</span><span class="token number">160</span><span class="token punctuation">,</span> <span class="token number">160</span><span class="token punctuation">,</span> <span class="token number">180</span><span class="token punctuation">,</span> <span class="token number">200</span><span class="token punctuation">]</span><span class="token punctuation">,</span> getLineColor<span class="token operator">:</span> <span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">,</span> getLineWidth<span class="token operator">:</span> <span class="token number">1</span><span class="token punctuation">,</span> lineWidthMinPixels<span class="token operator">:</span> <span class="token number">1</span><span class="token punctuation">,</span> pickable<span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span> stroked<span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span> filled<span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span> wireframe<span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre></div> <p>So we now have an instance of a polygon layer generated from the data read from the disk. The last thing to do now is add it to the Deck.gl component.</p> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">layers <span class="token operator">=</span> <span class="token punctuation">{</span> mapLayers <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre></div> <p>The result:</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/3736201b821bc00e808c92f36c992877/50383/polygon-deckgl-map.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 41.62162162162162%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAIAAAB2/0i6AAAACXBIWXMAAAsSAAALEgHS3X78AAABX0lEQVQY02WN3UvCUByG94dHUVYEQRF004VhZYRiYGXgF5W70CjnXLqNTXfcOWer3HC0zTU3tTquKMPn4uVwnvfHS22dNrfPuNgxGzti10+aOxlp/xLsZZXVBLN0cL8cf1hLMCuHj5tJjthZLWpuJLndc4lCiqh2BdgVEBANHVh9PA7dcTgEUqsjcL1OW+CYaoUWnhrkBwNR7fCkab6oloEprOnttijLXYjI81kBPZI9FTZYrlCsFAp0Op1PpUqZTDmbvalW60SpEEUdRCmKIoqiJEmqqmqaRlKWZZ7n8/lyPJ6+yN6WS7XrHJ27uisVaYZhif1tzo4BACQxxpZlmaaJI1i2VavVIUSGYZimgRCGCOq6BiFJfTAYvPb71JvtDCPCMPxcwHYc27bdoev7/qKlPM8PI4IgGI1Gk8nkYw7v3Q+CPzudTuctWf7BcRzP88jxv2Xb+bauOxsnx/P2C9pEogcW5KgPAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Screenshot of styled Deck.gl map with polgon layer" title="Screenshot of styled Deck.gl map with polgon layer" src="/static/3736201b821bc00e808c92f36c992877/50383/polygon-deckgl-map.png" srcset="/static/3736201b821bc00e808c92f36c992877/1d79a/polygon-deckgl-map.png 185w, /static/3736201b821bc00e808c92f36c992877/1efb2/polygon-deckgl-map.png 370w, /static/3736201b821bc00e808c92f36c992877/50383/polygon-deckgl-map.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>As you can see we now have a polygon layer displayed on our map but none of the additional information included in the geojson is visible to the user. Most of deck.gl’s layers can be configured to display tooltips on top of the layer. The <code>pickable</code> prop that I added to the PolygonLayer allows the deck.gl component to access all the additional information and adding a <code>getTooltip</code> prop to your deck allows you to manually configure the structure of your tooltip.</p> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">getTooltip<span class="token operator">=</span><span class="token punctuation">{</span><span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> object <span class="token punctuation">}</span></span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>object<span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">const</span> typedObj <span class="token operator">=</span> <span class="token punctuation">(</span>object <span class="token keyword">as</span> unknown<span class="token punctuation">)</span> <span class="token keyword">as</span> <span class="token punctuation">{</span> properties<span class="token operator">:</span> <span class="token punctuation">{</span> story<span class="token operator">?</span><span class="token operator">:</span> string<span class="token punctuation">;</span> scalerank<span class="token operator">?</span><span class="token operator">:</span> number<span class="token punctuation">;</span> sovereignt<span class="token operator">?</span><span class="token operator">:</span> string<span class="token punctuation">;</span> level<span class="token operator">?</span><span class="token operator">:</span> number<span class="token punctuation">;</span> type<span class="token operator">?</span><span class="token operator">:</span> string<span class="token punctuation">;</span> geounit<span class="token operator">?</span><span class="token operator">:</span> string<span class="token punctuation">;</span> subunit<span class="token operator">?</span><span class="token operator">:</span> string<span class="token punctuation">;</span> name<span class="token operator">?</span><span class="token operator">:</span> string<span class="token punctuation">;</span> name_long<span class="token operator">?</span><span class="token operator">:</span> string<span class="token punctuation">;</span> economy<span class="token operator">?</span><span class="token operator">:</span> string<span class="token punctuation">;</span> continent<span class="token operator">?</span><span class="token operator">:</span> string<span class="token punctuation">;</span> pop_est<span class="token operator">?</span><span class="token operator">:</span> number<span class="token punctuation">;</span> gdp_md_est<span class="token operator">?</span><span class="token operator">:</span> number<span class="token punctuation">;</span> income_grp<span class="token operator">?</span><span class="token operator">:</span> string <span class="token punctuation">}</span> <span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>typedObj<span class="token punctuation">.</span>properties<span class="token punctuation">.</span>continent<span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Name: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>typedObj<span class="token punctuation">.</span>properties<span class="token punctuation">.</span>name_long<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">\nContinent: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>typedObj<span class="token punctuation">.</span>properties<span class="token punctuation">.</span>continent<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">\nGDP: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>typedObj<span class="token punctuation">.</span>properties<span class="token punctuation">.</span>gdp_md_est<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">City: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>typedObj<span class="token punctuation">.</span>properties<span class="token punctuation">.</span>name<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">\nStory: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>typedObj<span class="token punctuation">.</span>properties<span class="token punctuation">.</span>story<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token keyword">null</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre></div> <p>Now you can see any specific data that you want when the user hovers over a certain section of the map. Without any need to specify coordinates, boundaries etc. Everything is handled for you.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/8865169afc59fdc82b29c6e3c7ebcef5/50383/tooltip-deckgl-map.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 41.08108108108108%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAIAAAB2/0i6AAAACXBIWXMAAAsSAAALEgHS3X78AAABbUlEQVQY02WO60vCUBjG938H0RWCCIr6ElHYjVCsRtmHqVgqVITLWWtjs23nnF3YnA11rV3U2ekYUfT78PCe93ke3kOtHHOrJ63Fg+bifnPpgFvLS5sX2gatzGfYme372d2HhQw7t9dYPmotHXJfMZLEz/VTiYKKCF4F+CogRXRM9c3Rh7E/TN5V6bktcFqbF1qPt9Wa8PSIN0gVtfYLVtcCnqNTumHyvCjLCkB4tBRVw6oBxLIcw1SZYj2XY7LZUj5fpunK7V1DAxBARDKQUhRFFEVJkgAAhmFglWWZf+Fvruv7mTP6vFIu3V0VaoXLm2Kxyjaa2P1JfpVVVcWq67rnea7rQoQcx97Y2jk6zlumhedOp4OQDiE0TQOrZZndbte2barX678TkiSZENI0xYo/5pM9CfhRFE7+QQUfYUKI4ziKotFolBKmdhD8ccfjcfoLqtf/ZjAYBEGAy9MaSU3weur6vh+GIS7/vvwJxgagpB38fpYAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Screenshot of styled Deck.gl map with polgon layer with tool tips" title="Screenshot of styled Deck.gl map with polgon layer with tool tips" src="/static/8865169afc59fdc82b29c6e3c7ebcef5/50383/tooltip-deckgl-map.png" srcset="/static/8865169afc59fdc82b29c6e3c7ebcef5/1d79a/tooltip-deckgl-map.png 185w, /static/8865169afc59fdc82b29c6e3c7ebcef5/1efb2/tooltip-deckgl-map.png 370w, /static/8865169afc59fdc82b29c6e3c7ebcef5/50383/tooltip-deckgl-map.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>The last feature we need to add to our map is an icon layer showing where the stories we have on our site come from. In the same saga I mentioned earlier we need to instantiate a new IconLayer class.</p> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"><span class="token comment">// Initial story data that could be pulled in from database</span> <span class="token keyword">const</span> pointData <span class="token operator">=</span> <span class="token punctuation">[</span> <span class="token punctuation">{</span> properties<span class="token operator">:</span> <span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token string">'Chavignol'</span><span class="token punctuation">,</span> story<span class="token operator">:</span> <span class="token string">'5 Cheese Hacks Only the Pros Know'</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> coordinates<span class="token operator">:</span> <span class="token punctuation">[</span><span class="token number">2.844560</span><span class="token punctuation">,</span> <span class="token number">47.397411</span><span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> properties<span class="token operator">:</span> <span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token string">'Milan'</span><span class="token punctuation">,</span> story<span class="token operator">:</span> <span class="token string">'11 Hottest Street Light Trends for 2022'</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> coordinates<span class="token operator">:</span> <span class="token punctuation">[</span><span class="token number">9.195585</span><span class="token punctuation">,</span> <span class="token number">45.467905</span><span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> properties<span class="token operator">:</span> <span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token string">'Kyiv'</span><span class="token punctuation">,</span> story<span class="token operator">:</span> <span class="token string">'9 Best Practices for Remote Workers in the Shark Industry'</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> coordinates<span class="token operator">:</span> <span class="token punctuation">[</span><span class="token number">30.479160</span><span class="token punctuation">,</span> <span class="token number">50.44044</span><span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> properties<span class="token operator">:</span> <span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token string">'Zurich'</span><span class="token punctuation">,</span> story<span class="token operator">:</span> <span class="token string">'10 Ways Investing in Maple Syrup Can Make You a Millionaire'</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> coordinates<span class="token operator">:</span> <span class="token punctuation">[</span><span class="token number">8.519453</span><span class="token punctuation">,</span> <span class="token number">47.385598</span><span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">]</span> <span class="token comment">// Icon layer based on the story data above</span> <span class="token keyword">const</span> layer <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">IconLayer</span><span class="token punctuation">(</span><span class="token punctuation">{</span> id<span class="token operator">:</span> <span class="token string">'icon-layer'</span><span class="token punctuation">,</span> data<span class="token operator">:</span> pointData<span class="token punctuation">,</span> pickable<span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span> iconAtlas<span class="token operator">:</span> <span class="token string">'https://upload.wikimedia.org/wikipedia/commons/e/ed/Map_pin_icon.svg'</span><span class="token punctuation">,</span> iconMapping<span class="token operator">:</span> <span class="token constant">ICON_MAPPING</span><span class="token punctuation">,</span> <span class="token function-variable function">getIcon</span><span class="token operator">:</span> <span class="token parameter">d</span> <span class="token operator">=></span> <span class="token string">'marker'</span><span class="token punctuation">,</span> sizeScale<span class="token operator">:</span> <span class="token number">15</span><span class="token punctuation">,</span> <span class="token function-variable function">getPosition</span><span class="token operator">:</span> <span class="token parameter">d</span> <span class="token operator">=></span> <span class="token punctuation">(</span>d <span class="token keyword">as</span> <span class="token punctuation">{</span> coordinates<span class="token operator">:</span> Position2D<span class="token punctuation">;</span> exits<span class="token operator">:</span> number <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">.</span>coordinates<span class="token punctuation">,</span> <span class="token function-variable function">getSize</span><span class="token operator">:</span> <span class="token parameter">d</span> <span class="token operator">=></span> <span class="token number">5</span><span class="token punctuation">,</span> <span class="token function-variable function">getColor</span><span class="token operator">:</span> <span class="token parameter">d</span> <span class="token operator">=></span> <span class="token punctuation">[</span><span class="token number">50</span><span class="token punctuation">,</span> <span class="token number">168</span><span class="token punctuation">,</span> <span class="token number">82</span><span class="token punctuation">,</span> <span class="token number">255</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre></div> <p>The result is a map containing information about both the countries we’ve specified and the stories on our site.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/a8d9e1d86a376d11123ffb6d3ece04ca/50383/point-layer-deckgl-map.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 41.08108108108108%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAIAAAB2/0i6AAAACXBIWXMAAAsSAAALEgHS3X78AAABaUlEQVQY02WN7UvCUBTG938HRWkgQUj4JSErLTFy4QtIFDHTD2b5spm2sbnr3u5E08Q259xm1tFChH4fHs49z/OcSwTizEGi4YvRvijtjzGHJB/KSkdptHte2wqXtyMvMOycVvYvG/4LZhmL0XtRGp7BG55QECeJrCKyKuL6euejr3mO6bkTxDfbLCO1W+xrvVQogsJG7XBS+w30vStDktCw3mpxgoBkFcYu6kigkqzSjSZFlfL5ciaTJ0mKJB/Saar0VJVkRVZUjCGjEAghjuN4npdlGWMMKggCy7K3qWwwEAofJ04iyUQil8sV7+8KtSoN7jq5LAOiKGqaNhqNBoOBqqpY15+px7NI9DqZil8lK9W6rnfXHYz14XDY6/WI8acxMSeA67rf/3Ac1/M8x5lNp9P/LjGxpu4KByKz2Xw+X6z4WgGubc/gxG8ANosN4Oc/DMOwLAvKm7cNwxyPx2CZpmnbNpQ33R8HhJ7mBUyhjgAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Screenshot of styled Deck.gl map with polgon layer with article points overlayed" title="Screenshot of styled Deck.gl map with polgon layer with article points overlayed" src="/static/a8d9e1d86a376d11123ffb6d3ece04ca/50383/point-layer-deckgl-map.png" srcset="/static/a8d9e1d86a376d11123ffb6d3ece04ca/1d79a/point-layer-deckgl-map.png 185w, /static/a8d9e1d86a376d11123ffb6d3ece04ca/1efb2/point-layer-deckgl-map.png 370w, /static/a8d9e1d86a376d11123ffb6d3ece04ca/50383/point-layer-deckgl-map.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <blockquote> <p><em>Note: In the real world we would have the icon layer pull from a database so that it’s automatically updated on publishing of a new blog.</em></p> </blockquote> <h2>Technical Deep Dive</h2> <p>You’ve now seen how easy it is to display rich maps using deck.gl and Carto. I want to take some time to talk about what makes them such interesting pieces of software. In the world of geospatial analysis there are some standard data formats (GeoJSON, Shapefile, KML, CSV, etc) and ArcGis is considered to be at the forefront of the geospatial data handlers. You can think of an ArcGIS server as blob storage with an API for processing and access (although in reality it’s A LOT more complex and feature rich). The ArcGIS server is configured to interpret and process data in a geospatial format and make it easy for developers and analysts to use it where appropriate. Carto can act as both an <a href="https://carto.com/blog/esri-and-cartodb-together/">intermediary</a> or a replacement for ArcGIS. But using it as part of your stack can vastly optimise your load times and improve developer/user experience due to the way it handles your geospatial data.</p> <h3>The Power of the_geom</h3> <p>Let’s talk through another example, you want to create a polygon layer to display on a map of Europe to show the most recent population of each country. To generate this you would need three things:</p> <ul> <li>A map of Europe - <a href="https://www.mapbox.com/">https://www.mapbox.com/</a></li> <li>The contour data defining the boundaries between countries with a relatively high degree of precision - <a href="https://www.kaggle.com/sudhirnl7/human-development-index-hdi">https://www.kaggle.com/sudhirnl7/human-development-index-hdi</a></li> <li>The population data tied to each of the contours for a country - <a href="https://www.kaggle.com/sudhirnl7/human-development-index-hdi">https://www.kaggle.com/sudhirnl7/human-development-index-hdi</a></li> </ul> <p>So now you have the data, where will you store it? Well for a single country you will likely have millions of data points outlining its boundaries which can amount to ~1GB of data for each country. There are 44 countries in Europe. Now you have to load ~44GB for the user each time they want to see this visualisation. Carto offers a great solution to this. Instead of storing the contour data for a country on a point-by-point basis, it takes the data you’ve scraped from the internet (GEOJSON for example), ingests it and converts it into what they call the <code>the_geom</code>. This compresses millions of data points into one, which can easily be queried, manipulated or converted back into the GeoJSON format all using standard SQL commands. <a href="https://carto.com/help/tutorials/getting-started-with-sql-in-carto/">Now you’ve gone from having to load millions of rows of data to only 44.</a></p> <p>So we have our data compressed and stored on the Carto servers which can easily be accessed using the <a href="https://carto.com/developers/sql-api/">Carto API</a>, now we need to figure out how to get it displayed on our map.</p> <h3>More tiles than the kitchen floor</h3> <p>On the homepage deck.gl states it’s a webgl powered framework for large scale datasets. Given how the size of geospatial datasets can quickly grow, having a library that can process and handle large volumes of data on the web is key to building an application with a good user experience. Some of the features I find most interesting are:</p> <ul> <li>Loading Data: Deck.gl is built on top of loaders.gl which is a library the same team built dedicated to handling the loading of large amounts of data. Providing loaders for geospatial, csv, json, vector, image files and more. Leveraging the loaders.gl library means that the load times of large datasets can be kept to a minimum with ease.</li> <li><a href="https://deck.gl/docs/developer-guide/building-apps">Bundle optimisation</a>: Another interesting trait of deck.gl is that it’s built with web developers in mind. Hence features like tree shaking can also be easily configured to keep your bundles as small as possible.</li> <li><a href="https://deck.gl/docs/developer-guide/64-bits">64-bit layers</a>: Given that there are so many use cases for deck.gl the software is built in such a way that it can handle visualisations for many scenarios. The use of emulated 64-bit floating point means that the precision and detail of the visualisations is incredibly high. This means if your data goes down to the centimeter scale then the user will have no issues zooming in to that level.</li> </ul> <h2>The End...?</h2> <p>So you've now seen how easy it is to add data rich map visualisations to your web applications. You can see all the code I used to build my demo app <a href="https://github.com/Eddie-MG/terravisionnews2021">here</a> and the link to the actual site <a href="https://terravision2021.herokuapp.com/">TerravisionNews 2021</a>. Any questions feel free to get in touch, PRs/comments welcome.</p><![CDATA[Comparison of Cloud Run and Lambda to render Blender scenes serverlessly]]>/2021/11/blender-serverless-cloud-comparison//2021/11/blender-serverless-cloud-comparison/Tue, 09 Nov 2021 00:00:00 GMT<p>In a <a href="/2021/08/blender-serverless-lambda/">recent post</a>, I explained how to run Blender render jobs on AWS Lambda functions, using Lambda container images. As a follow-up, today I am looking at the same workload running on Cloud Run, the easiest serverless way to run a container on Google Cloud Platform.</p> <h2>Test protocol</h2> <h3>vCPUs and RAM</h3> <p>A quick disclaimer is that if you can choose between GCP and AWS, in this case, AWS seems to be a better choice because your Lambda functions can use up to 6 vCPUs whereas Cloud Run is limited to 4. When rendering Blender scenes, increasing CPU power has a huge impact on rendering times. I included a Lambda function with 6 vCPUs in our benchmark and it's about 30% faster than running on 4 vCPU-solutions (be it on GCP or AWS).</p> <p>The topic of memory is interesting. With Lambda, you cannot independently choose the number of vCPUs and amount of RAM. Based on the amount of RAM you pick, the Lambda will be allocated a certain number of vCPUs. If you pick 10GB of RAM (the largest option available on Lambda) you will get 6 vCPUs. 6.8GB of RAM give you 4 vCPUs, so that's what I went for in order to have the same number of vCPUs on both Cloud Run and Lambda.</p> <p>With Cloud Run on the other hand, you can independently set vCPU count and RAM amount. The maximum amount of RAM available is 16GB which is significantly more generous than what Lambda offers. Depending on what scene you are willing to render, this can be a crucial point because larger 3D scenes in Blender will require more RAM. So it might happen that you simply cannot render the scene you want on a Lambda with 10GB of RAM. In the use-case that led us to set up this system, I was rendering very small scenes so 10GB were more than enough.</p> <h3>Container image</h3> <p>The container image I am using with Cloud Run is very similar to the one I used on Lambda and described in the <a href="/2021/08/blender-serverless-lambda/">previous post</a>. It is also based on the Ubuntu image with Blender that is maintained by the <a href="https://github.com/nytimes/rd-blender-docker">R&#x26;D team at the New York Times</a>. The main difference is that I do not install the AWS specific SDK and I run an HTTP server with Gunicorn. The impact of this server on the overall performance of the solution will be minimal since the large majority of the response time is spent rendering the Blender scene.</p> <h3>Region</h3> <p><a href="https://medium.com/google-cloud/cloud-run-and-cloud-functions-does-the-region-change-the-performances-b967e5cee0cc">This recent blog post</a> published by Guillaume Blaquiere shows that not all Google Cloud Platform regions are equal when it comes to Cloud Run performance. I had initially set up our experiment in the us-east4 region and, after reading this, I decided to add a test run in the fastest region. In his article, Guillaume Blaquiere finds out that his code runs 11% faster in the us-west3 region compared to us-east4. I got similar results.</p> <h3>Execution environment</h3> <p>Another variable I took into account is Cloud Run's execution environment. Google gives us the option to either run in a first or second generation execution environment. The second generation is in preview at the moment.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/110b52003a72f75d7e8dbc16907953d2/50383/gcp-cloudrun-execution-environment.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 26.486486486486488%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAFCAYAAABFA8wzAAAACXBIWXMAABYlAAAWJQFJUiTwAAAAyElEQVQY04WQawrCMBCEe3/wAP7yFv4UPEbro6U0aaU0TfpKWpKM6YoigrrhYycszE4STdMExhg456iqCkVRkF6WBd57OOeo0/H/iVbDNE2RZRmZ5XlOummaB6KBbBVUOULyAda634YINc8GbduirmtKaYyh4ZrymdDZB9/qzdCjEjOSK8P5lCBOEvDwBfvDEZvtDvHlCh9MtdFh8UzLPtFavxZH3rtwAYZhhBACUkriFtKysgzPFlBKEX3fo+s6YtVP1pm1lgzv5QSAz9/3VVAAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Screenshot of the Google Cloud Platform interface showing the two available options for Cloud Run&#39;s execution environment at the time of writing" title="Screenshot of the Google Cloud Platform interface showing the two available options for Cloud Run&#39;s execution environment at the time of writing" src="/static/110b52003a72f75d7e8dbc16907953d2/50383/gcp-cloudrun-execution-environment.png" srcset="/static/110b52003a72f75d7e8dbc16907953d2/1d79a/gcp-cloudrun-execution-environment.png 185w, /static/110b52003a72f75d7e8dbc16907953d2/1efb2/gcp-cloudrun-execution-environment.png 370w, /static/110b52003a72f75d7e8dbc16907953d2/50383/gcp-cloudrun-execution-environment.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h3>Test scene</h3> <p>For our test, I render a 500-pixel-wide square image of a test scene provided by Mike Pan and available on <a href="https://www.blender.org/download/demo-files/">Blender's Demo Files page</a>. I used Blender's Cycle rendering engine. You can see the output below.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 250px; " > <a class="gatsby-resp-image-link" href="/static/f70e6d4cc3349158d5682b5a4826b6d8/63868/render.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 100%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAACXBIWXMAAAsTAAALEwEAmpwYAAAD7UlEQVQ4y4WPSWwbZRTH58ABhKKkiSI3cRPi2LFje+wZLzOe8ZbxeImdxFvGY3vG+3gmHi+xHSd2EjeLUpRCEWJRCgL1QJQcy4FKnNoDnDhxRUgVB0SFlAPLBXGJRBhTKhUhivTT06fvfb//ex8A414T7oVfhO+/WgD8P+aLeCZjXggjpWrCBud/A/0N+TzAXzv7THa/1e6X2qCNNP7j9XOa7RkD0wPZPIDB5oUQwmZxuDFikQh6iaAV90G2pwxSTFL04IY0DvAMQAkjOqgAZvcmzdYMZmcQJAxCJZdnwezUmAkT5jfafHqEnIPdc5BTZyWe7i85BkRiQQJAQCs3r75FEo9y9JPDrftM0jtrhECHAQ/qEB/hXmJXWSZZcBHLc0bHjA7VmF2g1S2ht7gA2YRyeli2Mq3owqaHTPKOc5G8gbjH5xmzs7wUL2e4br11u793dnJyfnK3V2+73Itas1NvdurMDkBvwKZnwFHZ/CvDavcofIu5+XD3jYvze3988eDi0edffvbg9t4RFU4guGe/1Tp/9206FNaCiBp2aGEcWCAjHmKFdAYd6KJqyn1aO763eedA3Dn7+PS3H7+7urq8uvz9p8fffvrRh+f97ge9jYDLE7VYjTCmMqDAgi/m8cWC/nh8KUmAHt0QXLIEDoLxHYf/q3fe+/nil18f/3D5/TdP7p+dphLvF/N3c+m3UEPQAE/rEIAMUt4gFQhS/sCqUWefHFLKh+bmr2mSGryFhj8p7H7der0XzitlRnJCiY9PYePywOysQW2Y1SNAIJwORpjlKBtaSZpRUv4aNHZNNfKqYvglhfxlnWfElJ5Cr09hOqU1Mgd6Z9TYDYVOYwAtbi1kB5bi2TCVi1L5GF0Mx7O+QMxq86o06JgMnBjVqkc0qusGlRaTyzTqcQUyqVCMTYKwzekJ4+5lIJIoxJKl1TSXYIVUrkqny7HVbGiJJskwivk1oFOpts3rca0eByGn0eyCLU67a5HwRxd8USCWKlEsT2eEdL6a5ZrZYi1XrGVylXS6nEpxVDwTWqb9oUQ0kkrEWZrKJuhClMqtxNhQJAVQrJCUnubFDNfIC60C3+LW2rzYqVS3xHpP4JtsrpJieYYpMyzPZivpXIVmhVVpQboIJPMiU6hluUZhrV0SO+XqplDrVte3m539Tu+o2eqviZuc0OL4ZmkQ3cxxjXShKs1LsDyQHpjreUEyN/l6V2j0xOZuY+PmRu+o2z/ubB022v3a+rZY767Vu7y4WVxrZ8vrTKmWyosAI31SShU7fL1Xae5U2/1GZ6/dPdzqH+/sv9mT/O2j1tZBs7NXb/fF5o40oCh2ckKL5Rp/ArsEbjDZtiUnAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="The final render that is created by each of our test runs." title="The final render that is created by each of our test runs." src="/static/f70e6d4cc3349158d5682b5a4826b6d8/63868/render.png" srcset="/static/f70e6d4cc3349158d5682b5a4826b6d8/1d79a/render.png 185w, /static/f70e6d4cc3349158d5682b5a4826b6d8/63868/render.png 250w" sizes="(max-width: 250px) 100vw, 250px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h2>Results</h2> <table> <thead> <tr> <th>Test settings</th> <th>Specs</th> <th>Region</th> <th>Average duration of 4 function executions (s)</th> </tr> </thead> <tbody> <tr> <td>Cloud Run</td> <td>4 vCPUs, 16GB, first gen</td> <td>us-east4</td> <td>79.8</td> </tr> <tr> <td>Cloud Run</td> <td>4 vCPUs, 16GB, second gen</td> <td>us-east4</td> <td>81.4</td> </tr> <tr> <td>Cloud Run</td> <td>4 vCPUs, 16GB, first gen</td> <td>us-west3</td> <td>77.7</td> </tr> <tr> <td>Cloud Run</td> <td>4 vCPUs, 16GB, second gen</td> <td>us-west3</td> <td>71.5</td> </tr> <tr> <td>Lambda</td> <td>4 vCPUs, 6.8GB,</td> <td>us-east-1</td> <td>76.1</td> </tr> <tr> <td>Lambda</td> <td>6 vCPUs, 10GB,</td> <td>us-east-1</td> <td>51.5</td> </tr> </tbody> </table> <p>A few observations:</p> <ul> <li>The results with 4 vCPUs are all pretty similar</li> <li>More vCPUs means noticeably faster render times. That's not breaking news but in our case this really meant Lambda was the way to go.</li> <li>We can indeed see that Cloud Run does not perform in the same way in all regions. In this case, it performed 14% slower in us-east4 compared to us-west3.</li> <li>The second generation execution environment of Cloud Run seems to be significantly faster than the first in the us-west3 region but slightly slower in the us-east4 region. Given the fact that this new generation is in preview for now, we can expect things to get better in the future but, for now, test</li> <li>Using Cloud Run on 4 vCPUs in the us-west3 region is about 6% faster than Lambda with 4 vCPUs in the us-east-1 region.</li> </ul> <h2>Next steps</h2> <p>The impact of the region on Cloud Run performance made us wonder if there is a comparable impact on Lambda. So I may update this article with additional information on this.</p> <p>I would also like to measure the performance impact of using Graviton2 Lambda functions for our task.</p> <p>It would be interesting to see a serverless solution that lets us use GPUs as this would dramatically speed up rendering jobs. But would the cost of such a solution make it a better solution for our use case? I would have to try!</p> <p>I will post more article on the topic of serverless Blender in the future. <a href="https://twitter.com/jiherr">Follow me on Twitter</a> to receive updates!</p><![CDATA[How to Generate Beautiful PDFs with React and Puppeteer]]>/2021/10/pdf-generation-react-puppeteer//2021/10/pdf-generation-react-puppeteer/Wed, 27 Oct 2021 00:00:00 GMT<p>I recently had to provide a new functionality on my project: the "download as PDF" one. The first thing I wondered was why should we provide this functionality? Doesn't it already exist natively with pretty much all web browsers with a right mouse click / print / "save as pdf" option? Well I tried on my webpage and the result was really disappointing:</p> <ul> <li>I had page breaks in the middle of my data tables, graphs, or any React component which aren't supposed to be split on different pages</li> <li>I didn't have any title, header, page number except the ugly default ones provided by the browser options</li> <li>I had many other UX/UI pain points that made the PDF quite unreadable</li> </ul> <p>Example on a <a href="https://www.boursorama.com/bourse/opcvm/cours/composition/MP-468273/">Boursorama random page</a>, that looks like:</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/481e19cb4d7d0137356e4c76df115fef/50383/boursorama_example.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 129.7297297297297%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAaCAYAAAC3g3x9AAAACXBIWXMAAAsSAAALEgHS3X78AAAEWElEQVRIx41VW2tcVRQO+KLipVbwQZ8EFbQG9cnfIFVsM00zSWvUpsFqSVMlUkQQQXxSoVUk0TIKImkrNYq+BdHatLnMTG3UMU0mJpOZTOZ6zpwzZ8799rn2PnMmF0zawyzO2Xuv/e21vv2tNW2nP/0F33w1gfMXExj+dhpnzsdx+lwcn9H78wsz+JJs5OvfcXZ0ErGxBGLfJ3D2YhwjF6ZpbQofxiYQ+egS3nnvR3xw8Azabnv4eTz0bC+eeW4ADzzdjd17OnH3Y/uw6/EI7iG769EXcV97F3a3H6Tvfbj3iQO445EXcH97FLue7KQ9UTz4VBfuJL/b9xxA2/5Dx9B1ZAA9/W8iemQQ0VdPoKP3dURePo5Oso7Dx9DTdxLRvhPofOU4eo6eRITmon2DiPS+ge7+QXTTXEfPa9hP1gZ6DNuDpDkQGxaEhgmlpkIXVTRMFypZw3BgOx5zhef5oB98nw/h0oBZcxgA6qUKFhYyWC0KqFQl6JoFTTGgEpBm2FgsKhifq+JqRkGhbqGhmigICioSHaqZkHUbomqvA2orOcyl0sjmy6iWBGimQ9E5kAlYps0loY6VXBHFUpXWXAh0WJoOERQTpkVZGCwTZwNgvoj8moCC2IBITgoBVSQNRVGBbtrQKUpLkWBIAgwa1+iQGlHjEgsm0cVAHdfbkLJuokwp1Ooa1soyN0nRoeoWN5a27QIOEcU2ep7HfZlZtsNtE6BHDNcViq5Ww8JyAdN/ZpFMrSLxdw7xv3K4lspBaWh8g0++zFQKQlE1aLqxFdDngIZpQZJlpDMlTF5fwQyBMuCp2Szm/i3Atu1NgJwK2mMTmOu6/PbXI2RSIFNVlQMyIBbdDEXHIq3JSgsszCj8Dp9w3AT0uK5UTcPc4houJ5cx+UcGV65lMDObITrU/wdsvgML1lscGoZJ+lJRKEuYXypjfrmMG/ReylaJI3sToL8hwq2RBpVCfFRrpCsSqtww+Dh01gyLyDe5gB3iyiXy2QXsCGg7LhFMVUEpMwsdbdKKxmRD5oXpeh7ne1tANsGdfRdF2cXo1WX8NpXiiw4dFG7Il0QUq3KzfneM0G9qyMfYRBnRd3/CwPsxvsj0xR6mzUNvDeOloS+wlCu3QLdNOVwUicPh0UtIpVf52LQCwMvxeezt/wR7j36MqeuLQfSut33KXKikfN/3QlVxroIyCzaMjSfxw3ii1cJ25JA9FdngZVSiXijVVTgkFcZh0P/8TQL2dpKN35xsWDYB6tDZ2zAIzGnKhBqsSvOkU1UzgjK7mQ510lqFWlWF2heXCO/CHuewTL1QlFXeE1kbczkV7s6ATKysxbOo6qrRWmS3nMuvIfXPHEoVarzE8005ZBfgcuJ93MhqODWSxLmfrzRrFkgkk3h7aAiSJFPq2q0ArktgNi3g8KlRxL77FUFJWhDFGgRBoF5J/zWGcWuAYftiUQpU06ETO8iiS2LNgfVDdiFb29dWwP8Abe5whc7+dGAAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="boursorama example" title="boursorama example" src="/static/481e19cb4d7d0137356e4c76df115fef/50383/boursorama_example.png" srcset="/static/481e19cb4d7d0137356e4c76df115fef/1d79a/boursorama_example.png 185w, /static/481e19cb4d7d0137356e4c76df115fef/1efb2/boursorama_example.png 370w, /static/481e19cb4d7d0137356e4c76df115fef/50383/boursorama_example.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>rendering:</p> <figure class="myCustomClassName"><div class="myCustomClassName-grid" style=" grid-template-columns: repeat(auto-fill, minmax(31%, 1fr)); "> <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/e372af3bfa0a63496459eb1f73449e5d/50383/boursorama-page-1.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 140%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAcCAYAAABh2p9gAAAACXBIWXMAABYlAAAWJQFJUiTwAAAB+ElEQVRIx+1V2W7bMBD0//9Q+5ACBQrnwQLqHpJlxVbi6vapyxJFXdMlE6VKbNk9HpsBBkuuyOEulxRH4/EYhmE8U9M0qKoKlexsrhPnmOk6dGEldWjUV2ea9Avfdxp7t1hAURSMVqsV+mjb9rFRVQAvgZJYcLQdWYE6Z9K2RSFtQ32BIAgwWpCyQF3XUqwTPNg+NosHxJaPxAmQ+9tnsl5bMHM3qMMUlmVhtFwupUDTNC8E7x4sfDFM2N4aHk3I6hZZ1TxStInHqkVJ41ndgPKAbdungl3aRVkh5xV4VaMkEU6TONmCKPwZsaRvZsjgJbQFQ4Kd6GYXU3Q7eOsQTnCA5e2lDcgfpQxhWiBnHF/XR9yH+XXBOMmxOySIkgyHKMV2H2NP+5RmTO53FCcIoxgpWW3u4t4+nAqeVPoC0uMRm+0WP6hg7z7qUL45lyPs2ucoUNIeJ0kii/bh1sB05p2PcCi61wsJMMak4M3YwGfNvVzlc1H1+5wOfRTHsNwA7z/NoajOlWPDOVWRgYnb0PPjKQFON0iIChtRAQte/3vKr/FXKfcX6NgFM3hs/hTdQm+Cb4L/h+C1f+C5O9z/LgW7Z/RShN19HaJ4EgQs2/r10J/72/Qv/u/A932MJpOJDFWkbprmCwqfoHjAHccZpJjvui6m0yl+AuMNfvDpJQCdAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="boursorama_to_pdf1" title="boursorama_to_pdf1" src="/static/e372af3bfa0a63496459eb1f73449e5d/50383/boursorama-page-1.png" srcset="/static/e372af3bfa0a63496459eb1f73449e5d/1d79a/boursorama-page-1.png 185w, /static/e372af3bfa0a63496459eb1f73449e5d/1efb2/boursorama-page-1.png 370w, /static/e372af3bfa0a63496459eb1f73449e5d/50383/boursorama-page-1.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span> <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/0d338b63a2f9c60aedd9d10572dfef0c/50383/boursorama-page-2.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 140.54054054054052%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAcCAYAAABh2p9gAAAACXBIWXMAABYlAAAWJQFJUiTwAAACSklEQVRIx6VWy27TQBT1B3XHDhZ8Ae2GFY8PQOy6YR8hFlXFgg/gsWFDJVgiilSEQAKBiFJk8mhkFDeOHZsqbRx74lcOM+NMcOrxI2Kko5lrX585M/feGSvNZhP9fh+apvG+0+mg3W7zvtvtotfrrfUMwkfYbMy+bx23oFiWhaq2YFhk7KyRsR3HgTIcDrmRJAl/kQVrURTRl9HSZ5HzyfqOzBEUwzBWs8icgoDgx6kNtV/sJ3xNy1wnzC6BmwnBu2MfjaMpbtx/AvVktKY0O0EhoXCIk9R+9UHHzu4bXL+7h2ZbzxFmv61QmNox3cODwxY+fj9ZvisOSqlC1sg84MESLY5jrk4mQEp4eZN/GxMalDmsMxen1gRRGFA7yPmLSSoV2n6IkBJ6YYSJ59Plh1RlVJivUoVifH4xg/lnAptiOiMrzPwAHgnhzf9hRgKEdGdMs4TQcibQDQcDiqF5xolIEMGnBH7AEK3ASKOFhFBWVvkiLH5euuSs/YtWiWGfF5ZfaVDEOF6my/PXn7G18xBXb+3jS0tb1T02T+x0fO32Hh6/eI87D57iXuNlemDEyeaVIhTuPzvE1nYDV24+wttPaprgSVI/sWVn3tHXLr791KT7W6v0CmNc4FOpUIzdmY+p68H1CC5cn1eJqO2Nlsz6IKRV4BP4hFCihOOyz38vuepOqZXYRcf9Rmkji3odxWuEsluvTJVsYk5Ydo1micomFJE32TVaddHXXTY/kB0biqqq/BzTdT2HwWDAMR6PYdt2KdhfA/sl+QuCQ3Shc+/SowAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="boursorama_to_pdf2" title="boursorama_to_pdf2" src="/static/0d338b63a2f9c60aedd9d10572dfef0c/50383/boursorama-page-2.png" srcset="/static/0d338b63a2f9c60aedd9d10572dfef0c/1d79a/boursorama-page-2.png 185w, /static/0d338b63a2f9c60aedd9d10572dfef0c/1efb2/boursorama-page-2.png 370w, /static/0d338b63a2f9c60aedd9d10572dfef0c/50383/boursorama-page-2.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span> <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/0366ed4e1acc5d6d7b990513682865f7/50383/boursorama-page-3.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 140.54054054054052%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAcCAYAAABh2p9gAAAACXBIWXMAABYlAAAWJQFJUiTwAAAB7UlEQVRIx+1Wy46bMBTlC2cx3XfR9fxBt/MFI/UXZtcf6LJSpC4nnSjKQyRAmwQCQyYB/ACZU9tJhpDEkIzaXa90ZIzN8X0c21jdbhe2bcNxHEwmEwyHQwwGA92ORiOMx+Naq6DGDsfV/Ol0il6vBysIArRZuUObRVEEazab6Y4QAmVZHmH7fk8nTsYrKPN9H9Z8Pt96cW6SRJ4X+PYU4seza5y3JwyCI8K3EOWzENtAv/98waeHIW7uvsD2lrtoKpLLCXf9gRPj4+evuLt/RPya6kUOSQ6/NRLiYHVl8ZqA0Lx6V9ZLdJGHyjLKEYSrt6IwxmuhthIeT35NczizCIwS7RSldKeGenG2+TZ4uJeLMlJIKcQJck5ltXMJ3qjDRg/XmxRhvIYvQ94kBCnhEkynIZP5zFiFVPZzKVffRKhC8H4HmLgLTD0fthvg1/xFk1FegEgSwhUKDUVaNBMKECK9SlNkEptkI8dE69Y7G/IeSZLg9vYDOp0OMkKlbKhxp4i2KisPlXf9fh/L5RLxagXG+aler5ENpUznkjKGRJIz2V5NaBKsqX+Vh6YT5d0h/xXC99p/wn9EeP7Wu6zCNcLma7RZRtUBK6prtO2ivyYVURTCUgfAYrGA53kncF1XQy0ahqER6gBRrfol+QOC34Ytw+Ag8gAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="boursorama_to_pdf3" title="boursorama_to_pdf3" src="/static/0366ed4e1acc5d6d7b990513682865f7/50383/boursorama-page-3.png" srcset="/static/0366ed4e1acc5d6d7b990513682865f7/1d79a/boursorama-page-3.png 185w, /static/0366ed4e1acc5d6d7b990513682865f7/1efb2/boursorama-page-3.png 370w, /static/0366ed4e1acc5d6d7b990513682865f7/50383/boursorama-page-3.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span> </div></figure> <p>That's when I said "ok this feature may be worth it", but how should I do it? There are many open source libraries that can generate PDFs. But my choice went naturally to the well-known, google-developed library: Puppeteer. According to me, it is the easiest way to deal with PDF generation of Single Page Applications. It may not be so if you don't deal with javascript bundles but with plain HTML/CSS. Indeed, there are easier solutions for this use case like wkhtmltopdf, html-pdf-node or jspdf for example.</p> <p>In this article, I want to give you a few tips to generate beautiful PDFs of SPAs with Puppeteer. Firstly, I will explain to you how you can create a printable version of your page with React and Puppeteer. Then, I will show you how to use Puppeteer for the generation of your new printable page.</p> <h2>Render a printable version of your page</h2> <p>For this part you don't actually need to have Puppeteer or any other printer service set up. You can make your changes to your code as usual, and then ctrl+P on your page to see what it looks likes:</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/b50b31b476c12658efc1123165b6475a/50383/boursorama_print_chrome.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 66.48648648648648%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsSAAALEgHS3X78AAAB1UlEQVQ4y42TS44TMRCGcydOwYp7cAiOwIotD4kdLJkNmg1IbEAsQMwkE2ZYzERJOp1+u912d9v+qXKPm06ikSipVCrb9dXv1+znxRW2UYwkF1jvSqzjEpt9iaxqUNQKOXnVdMiLCovlH6RZgTwv0HYdrHVwzoFNygaVEJjNr64R7fZQrUFZCdQ00fWO3MDSYmOHAh5fLG8ITM2yDM5aaK2RpBnlOVKKnM940ZaAbKvc4M35JebLW5+zArgAlCOQC9kURc4FzYlaetUeuIkG4MdvOzx++hrvzr743JAK5w4VFgRQSlEzi7aloygp1y1ko/4Bo3uFSmn8mN/SRO9zhk2BcwaWFZ2XREfFfd97ZR3F9j73wHifoKpbvxXV9mhIgTFmhB4oJGBd1ygphrlgrHrGXeM4gdA93VKNXDRUIGGPgHyLfIGsKBSHHUydbnnYcpLXuNtmiFIBqTpIaiBbdgNtnAdNgaHR9Fg8MJxhb6z3Uf6k63iGR8Ap6AQ4FCl8+rqgqE+K5APA0afA9WbnF716/xmPnjzDS4r+2Rj7XwqHfALkr8d28XuFF2/P8Wu5OtjKQwpPwRNgeCbB+E3xGEe+Uf6nx0D+SbozuLxTeP4hxdl3gb/GdePPazA6KwAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="boursorama_print_chrome" title="boursorama_print_chrome" src="/static/b50b31b476c12658efc1123165b6475a/50383/boursorama_print_chrome.png" srcset="/static/b50b31b476c12658efc1123165b6475a/1d79a/boursorama_print_chrome.png 185w, /static/b50b31b476c12658efc1123165b6475a/1efb2/boursorama_print_chrome.png 370w, /static/b50b31b476c12658efc1123165b6475a/50383/boursorama_print_chrome.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>However, the feedback loop is not as quick as usual.</p> <p>To adapt your code for printing, you have to bypass the 2 main differences between a web page and a PDF:</p> <ul> <li>The components of a React application are dynamic whereas a PDF is a static file</li> <li>A PDF has page breaks and a fixed size whereas a webpage is a "one page" app with a variable viewport size</li> </ul> <h3>From dynamic webpage to static rendering</h3> <p>Create the "printable version" of your SPA with <strong>React.</strong> To create the printable version of our page, you will have to add/remove/modify the different components that make up the page.</p> <p>You basically have 2 solutions for this part:</p> <ul> <li>create a whole new custom design for your page</li> <li>adapt your current page</li> </ul> <p>If you opt for the second solution (that is less costly), you will have to adapt your existing components. For example, if you have a table with 3 tabs, you will probably want to display the content of all the tabs. Something like displaying the tabs one after the other may do the trick:</p> <p>Only dynamic:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">Table</span></span> <span class="token attr-name">selectedTabIndex</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>tab1<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span></code></pre></div> <p>Dynamic and Static:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">const</span> tabNames <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">'tab1'</span><span class="token punctuation">,</span> <span class="token string">'tab2'</span><span class="token punctuation">,</span> <span class="token string">'tab3'</span><span class="token punctuation">]</span> <span class="token punctuation">(</span>isPrintable <span class="token operator">?</span> tabNames<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">tabName</span> <span class="token operator">=></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">Table</span></span> <span class="token attr-name">key</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>tabName<span class="token punctuation">}</span></span> <span class="token attr-name">selectedTab</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>tabName<span class="token punctuation">}</span></span> <span class="token punctuation">/></span></span> <span class="token operator">:</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">Table</span></span> <span class="token attr-name">selectedTabIndex</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">'</span>tab1<span class="token punctuation">'</span></span> <span class="token punctuation">/></span></span> <span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre></div> <p>In this case, the <code>isPrintable</code> props will determine whether to display the 3 tabs, or just the first one. You may pass this props to every dynamic component of your page, that needs to be adapted for printing.</p> <h3>Deal with page breaks and fixed size with CSS</h3> <p>As you can see with the Boursorama example, your components may be cut off between 2 pages when trying to print your page. It happens because your web browser has no idea where to break page if you don't tell him. This is where the <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/break-inside"><code>break-inside</code></a> CSS property steps in. You obviously don't want your previous set of tabs to be cut off in the middle. Neither your graphs or almost any component on your page. Then you would have to adapt the previous code to add this CSS property. It would work with inline-css but you probably don't want to add the <code>style={{ breakInside: 'avoid' }}</code> everywhere in your jsx/tsx files.</p> <p>You would rather use stylesheets. And instead of adding this property on every CSS class already existing, you'll want to use the <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/@media"><code>media @print</code></a> option. This will let you custom your webpage for printing only! For example, you may want your text to be a bit bigger or to have a smooth grey color on the printable version, for any esthetic reason or convenience.</p> <p>We'll just add this in the <code>@media object</code> in your css file:</p> <div class="gatsby-highlight" data-language="css"><pre class="language-css"><code class="language-css">media <span class="token atrule"><span class="token rule">@print</span></span> <span class="token punctuation">{</span> <span class="token selector">body:</span> <span class="token punctuation">{</span> <span class="token property">font-size</span><span class="token punctuation">:</span> <span class="token string">"16px"</span><span class="token punctuation">;</span> <span class="token property">color</span><span class="token punctuation">:</span> <span class="token string">"lightgrey"</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token selector">.no-break-inside</span> <span class="token punctuation">{</span> // apply this class to every component that shouldn't be cut off between to pages of your PDF <span class="token property">break-inside</span><span class="token punctuation">:</span> <span class="token string">"avoid"</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token selector">.break-before</span> <span class="token punctuation">{</span> // apply this class to every component that should always display on next page <span class="token property">break-before</span><span class="token punctuation">:</span> <span class="token string">"always"</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span></code></pre></div> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">MyComponent</span></span> <span class="token attr-name">isPrintable</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span>true</span> <span class="token attr-name">className</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>no-break-inside<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span></code></pre></div> <p>These few CSS tips should help you improve a lot the rendering of your webpage.</p> <h2>How to use Puppeteer to generate your PDFs</h2> <p>Now, your page is ready for printing. You know it when you pass the <code>isPrintable</code> props to your page, right click + print on your browser, and you are quite comfortable with what you seeing. Here comes the part of printing. You now have a printable version of your webpage, but the users have no idea of it, and even if the ctrl + P on the website, they will see the "dynamic" version of the webpage. To let them generate the PDF version and automate the generation of the latest, you probably want to add a button that will directly generate the PDF server side, and even add some customization. This is what, among other things, Puppeteer is used for.</p> <h3>How Puppeteer works?</h3> <figure> <blockquote cite="https://www.imperva.com/blog/headless-chrome-devops-love-it-so-do-hackers-heres-why"> <p>Puppeteer is a common and natural way to control Chrome. It provides full access to browser features and, most importantly, can run Chrome in fully headless mode on a remote server [...]</p> </blockquote> <figcaption>—Dima Bekerman, <cite>https://www.imperva.com/blog/headless-chrome-devops-love-it-so-do-hackers-heres-why/</cite></figcaption> </figure> <table> <thead> <tr> <th align="center"><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/910d7d4aca7ebf5a016df6516792cdae/50383/how_puppeteer_works.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 41.62162162162162%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAYAAAD5nd/tAAAACXBIWXMAAAsSAAALEgHS3X78AAABNElEQVQoz1VRSW6EQAzkaTw19/wgx+TAEyIl0jBo2Ak009DNToVypkfEkuVFdrlse/u+g7pOE4aiwLZtoDBH/xxT1nVF3/foug7W2qdORz/Fc4XXPMHH2wsm2yFJUmRZJk3UMyAb7/c7TG8e+VUsQTnMW5YF8e0G8tBDh1YpJGmK9NC2baG1xlnGcRTWqtfQxsAMG+Z5FkDmvSAI4Pu+NKpG/VuFqzVNI4WcLoAHw33b8RV+4/MS4/VdIbxGMnw7tvDKsgRBh2EQkDiOcQlDRFEkLAlIMK5Kyzr69mBXVT9Q2khsjlgYulUYEJAMq6qSYk7lvQjknsEBzNV1LUPLIpftnoDum66Bmuc5yJyAbOSNyIK1vCHv7tjaB2P68hT3PVrXSOt8Np+FeQ7lNmT1p1YAifELfy1oFJDDjzMAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="how_puppeteer_works" title="how_puppeteer_works" src="/static/910d7d4aca7ebf5a016df6516792cdae/50383/how_puppeteer_works.png" srcset="/static/910d7d4aca7ebf5a016df6516792cdae/1d79a/how_puppeteer_works.png 185w, /static/910d7d4aca7ebf5a016df6516792cdae/1efb2/how_puppeteer_works.png 370w, /static/910d7d4aca7ebf5a016df6516792cdae/50383/how_puppeteer_works.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></th> </tr> </thead> <tbody> <tr> <td align="center"><em>Schema of how Puppeteer works server side</em></td> </tr> </tbody> </table> <p>Generation of the React app is done by a web browser. We need the minimal environnement able to execute javascript to render a DOM. Puppeteer will do it by launching a headless chromium. From now on, and since the generation is done on the server, the web browser doesn't need to have a graphical user interface (GUI). Chromium with generate the printable version: the same page the user sees on his web browser but with the <code>isPrintable</code> props activated. Then Puppeteer will execute the <code>pdf</code> function on the page with some custom options that will trigger the printing of the page.</p> <p>Just add the button with the URL that calls the printer service:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">Button</span></span> <span class="token attr-name">onClick</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>window<span class="token punctuation">.</span><span class="token function">open</span><span class="token punctuation">(</span>downloadUrl<span class="token punctuation">,</span> <span class="token string">"_blank"</span><span class="token punctuation">)</span><span class="token punctuation">}</span></span><span class="token punctuation">></span></span><span class="token plain-text">Download as PDF</span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span><span class="token class-name">Button</span></span><span class="token punctuation">></span></span></code></pre></div> <p>The <code>downloadUrl</code> is actually a GET request on your server that will execute Puppeteer on the server and return content with content-type <code>application/pdf</code></p> <p>So what does this Puppeteer code look like?</p> <h3>How to use it?</h3> <p>To be able to actually download the PDF, you just need a few code lines.</p> <p>The minimal code would then look like:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">const</span> puppeteer <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"puppeteer"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">(</span><span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> browser <span class="token operator">=</span> <span class="token keyword">await</span> puppeteer<span class="token punctuation">.</span><span class="token function">launch</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// launch a browser (chromium by default but you can chose another one)</span> <span class="token keyword">const</span> page <span class="token operator">=</span> <span class="token keyword">await</span> browser<span class="token punctuation">.</span><span class="token function">newPage</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// open a page in the browser</span> <span class="token keyword">await</span> page<span class="token punctuation">.</span><span class="token function">goto</span><span class="token punctuation">(</span><span class="token string">"https://printable-version-of-my-wbe-page.com"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> waitUntil<span class="token operator">:</span> <span class="token string">"networkidle2"</span><span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// visit the printable version of your page</span> <span class="token keyword">await</span> page<span class="token punctuation">.</span><span class="token function">pdf</span><span class="token punctuation">(</span><span class="token punctuation">{</span> format<span class="token operator">:</span> <span class="token string">"a4"</span><span class="token punctuation">,</span> path<span class="token operator">:</span> <span class="token string">"./my_file.pdf"</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// generate the PDF 🎉</span> <span class="token keyword">await</span> browser<span class="token punctuation">.</span><span class="token function">close</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// don't forget to close the browser. Otherwise, it may cause performances issues or the server may even crash..</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre></div> <p>These are the common steps you'll need to generate the PDF. Depending on your backend, you probably don't want not to download the PDF on the server but to render it on a response object, to send it back to the client (the web browser of the user). You should then adapt the <code>page.pdf()</code> method with <code>const buffer = await page.pdf({ format: 'a4'});</code> and return this buffer on the <code>_blank</code> page the user opened on his browser, waiting for a response.</p> <h4>Add some options to customize the PDF</h4> <p>You can of course adapt the options your naturally have on your browser, like the paper size, the scale, the margins, etc. with the help of the official documentation: <a href="https://pptr.dev/#?product=Puppeteer&#x26;version=v10.4.0&#x26;show=api-pagepdfoptions">https://github.com/puppeteer/puppeteer/blob/v10.4.0/docs/api.md#pagepdfoptions</a>.</p> <p>One cool option that I recommend, mainly because the default one provided by Google Chrome is really ugly, is the header or footer template. Just read a HTML file template and pass it through the data you want to display such as the current date, the page number for each page, or even an image/logo:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">const</span> footerBase <span class="token operator">=</span> fs<span class="token punctuation">.</span><span class="token function">readFileSync</span><span class="token punctuation">(</span><span class="token string">"./footer.html"</span><span class="token punctuation">,</span> <span class="token string">"utf8"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> customFooter <span class="token operator">=</span> footerBase <span class="token punctuation">.</span><span class="token function">replace</span><span class="token punctuation">(</span><span class="token string">"{{date}}"</span><span class="token punctuation">,</span> <span class="token keyword">new</span> <span class="token class-name">Date</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">.</span><span class="token function">replace</span><span class="token punctuation">(</span><span class="token string">"{{image_data}}"</span><span class="token punctuation">,</span> imageData<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">await</span> page<span class="token punctuation">.</span><span class="token function">pdf</span><span class="token punctuation">(</span><span class="token punctuation">{</span> format<span class="token operator">:</span> <span class="token string">"a4"</span><span class="token punctuation">,</span> footerTemplate<span class="token operator">:</span> customFooter <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre></div> <p>using a html template</p> <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>style</span><span class="token punctuation">></span></span><span class="token style"><span class="token language-css"> <span class="token selector">#logo</span> <span class="token punctuation">{</span> <span class="token property">height</span><span class="token punctuation">:</span> 40px<span class="token punctuation">;</span> <span class="token property">content</span><span class="token punctuation">:</span> <span class="token url"><span class="token function">url</span><span class="token punctuation">(</span><span class="token string url">"data:image/png;base64,{{image_data}}"</span><span class="token punctuation">)</span></span><span class="token punctuation">;</span> <span class="token punctuation">}</span> </span></span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>style</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>div</span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>body<span class="token punctuation">"</span></span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>div</span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>page-number-paragraph<span class="token punctuation">"</span></span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>span</span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>date<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>{{date}}<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>span</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>span</span><span class="token punctuation">></span></span>Page<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>span</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>span</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>pageNumber<span class="token punctuation">"</span></span><span class="token punctuation">/></span></span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>span</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>span</span><span class="token punctuation">></span></span>/<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>span</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>span</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>totalPages<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>span</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>div</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>div</span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>brand-container<span class="token punctuation">"</span></span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>span</span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>logo<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>span</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>div</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>div</span><span class="token punctuation">></span></span></code></pre></div> <p>You now have provided to your PDF a fully customized footer.</p> <p>There are a lot of other options regarding, the PDF generation, but also for the previous steps of launching the browser, opening a new page, going to the URL, that will let you fully customize your PDF generation on the server.</p> <h2>Conclusion</h2> <p>Finally, by adapting your React/CSS code and using Puppeteer, you can easily provide a fully custom PDF of your page. Moreover, Puppeteer is doing all the stuff server side. Which makes this feature fully transparent, quite fast for the end user, and with the same result for every user on any browser! Puppeteer is really powerful and has a lot of options that make the PDF generation quite easy for the developers, and with a rendering much more custom and beautiful than the default one on users' browsers.</p><![CDATA[A Quick Intro to Elm for React Developers]]>/2021/10/intro-to-elm-for-react-devs//2021/10/intro-to-elm-for-react-devs/Mon, 25 Oct 2021 00:00:00 GMT<p><em>Elm is probably my favourite programming language - its opinionated but welcoming design provides a compelling case study in how language design can gently guide developers towards writing maintainable code.</em></p> <h2>Why would a React developer learn Elm?</h2> <p>Elm is a language designed for creating frontend applications, offering an alternative to JavaScript libraries like React. Both were created circa 2012, Elm as an academic thesis project, and React as an industry-backed library. The underlying technology to both is a virtual DOM and a declarative programming style for creating user interfaces. The main difference between the two is that, while React is simply a library built for JavaScript, Elm is an entire domain-specific language, designed from the ground up for purpose.</p> <p>This means Elm has some language features which are now considered best practice in JavaScript development, such as immutability, lambda functions and type safety. But Elm goes further than React and TypeScript in many ways, with such a powerful type system that it can boast 'No runtime errors in practice', and a significant speed improvement over its competitors.</p> <p>As TypeScript gains more popularity, I wonder which other Elm features will move into React. My personal hope is that the famously friendly error messages (see image below) will make their way into the language. Elm is a compiled language, so the compiler is re-run after code changes to generate JavaScript for the browser. While this seems like a hindrance, I find the development loop quite engaging: a kind of 'type-driven-development' where the compiler helps guide your work, and gives the developer greater confidence in their code. This 'live-recompile' is made possible due to Elm's lightning-fast typechecking and compiling, something TypeScript is still sorely lacking.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/bebd9679c19bd9bca5de92353d20e9db/8efc2/compiler.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 49.72972972972973%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAYAAAC0VX7mAAAACXBIWXMAABYlAAAWJQFJUiTwAAABnElEQVQoz21SDXOrIBD0///G1yQm0cRPEAERxe0eTfo6nTJzA9zHsrdcEbcNntb5BWoJaLmL+bhh3Xase0LgHpiz7vu3yT2mhIWxcVkwsVas8CFAW4t+thitw20YUY0KM4MurFDzDMMHLO/G+2yauW6VmMUwmUzEEtjykWIYBrRNg6Hv8376+IfyfEbXttBa43a7oX2dq6rC8/FAWZaY+dCV+4W5noQSWR9kXGg9YefFzAZd1+VixeK2rlGXF4x85ElQRf/jekV1OaO532EJoqcpAwupnoQ82RfGGNxzgieDOidIsCf4nUDCoGJcGOA4Motscn7Zz1V8OROBSoLWZNhhpa6yEmOJxelX0Xv9BH1boZTCROpK9Wy5+dZN9nEcXzGFB7Wr6/8diP8v0CKQzcofkyTRZY/UUxsE/lzwjC0rFv7y9NJL9BatpOavlQGdc2govhS45OAPh3DQnzgeu83nbYuQXNF84dxJjTwi4HKW2hjj19icTic4zmA/sO3Qolme0JE/HZpsw9rDWJMlkLbf0yBn8Ul3QkjAPwHJQAEI3ohh/QAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="A compiler error" title="A compiler error" src="/static/bebd9679c19bd9bca5de92353d20e9db/50383/compiler.png" srcset="/static/bebd9679c19bd9bca5de92353d20e9db/1d79a/compiler.png 185w, /static/bebd9679c19bd9bca5de92353d20e9db/1efb2/compiler.png 370w, /static/bebd9679c19bd9bca5de92353d20e9db/50383/compiler.png 740w, /static/bebd9679c19bd9bca5de92353d20e9db/8efc2/compiler.png 828w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>The history of programming language adoption shows us that what's popular isn't always best, as the trending languages of today are often <a href="https://stackoverflow.blog/2020/09/02/if-everyone-hates-it-why-is-oop-still-so-widely-spread/#:~:text=the%20programming%20world.%E2%80%9D%C2%A0%C2%A0-,Was%20the%20success%20just%20a%20coincidence%3F%C2%A0,-OOP%20is%20still">successful by pure coincidence</a> - or as a result of corporate marketing campaigns. I truly believe the industry is moving towards functional programming: React's creator, Jordan Walke, said himself that ReasonML (a functional language closely related to Elm) is <a href="https://www.youtube.com/watch?v=5fG_lyNuEAw">'the language of React'</a>. He posits that, after the success of React, we should once again turn to functional programming and ask <em>'What else did we miss?'</em></p> <h2>An Example Project</h2> <p>To help developers familiar with React get to grips with the syntax, I've written the same application in Elm and React, a counter with an increment and decrement button (I've excluded a bit of boilerplate code and import statements from both to simplify things). The page looks like this:</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 326px; " > <a class="gatsby-resp-image-link" href="/static/3a082a791b4ba594ce6e857b3c8e8c37/ce9b1/counter.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 46.48648648648649%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAYAAAAywQxIAAAACXBIWXMAABYlAAAWJQFJUiTwAAAA8UlEQVQoz62SW4uDMBCF+/9/ki9aEXwXBUVBWK/BO+IV9ezOQKCt29JCAweTePLNnJDLcRx4Nugfad93vOMjXeRG0zSoqgp1XbPWdT0dHIaBPbcax/EOysBlWRCGIUNJcRyjbVsG0NpxHP4KkSPPc3RdxxJCIEmSM3CeZ4ZkWYYoitg4TRMsy4KmaTAMA6ZpMqTve+jXK3Rd54JPgVQ5CALYto2yLBnoeR5UVeXDruuiKAqOqCgKizxpmv4P9H2fQXQvMj51Q4VkZOqGksh7pvnPX6ITcNs2rk5AkuzkcVDkRx/t3QFfP4f982dzu/iGfgEtO7QVgHyIegAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="The counter application" title="The counter application" src="/static/3a082a791b4ba594ce6e857b3c8e8c37/ce9b1/counter.png" srcset="/static/3a082a791b4ba594ce6e857b3c8e8c37/1d79a/counter.png 185w, /static/3a082a791b4ba594ce6e857b3c8e8c37/ce9b1/counter.png 326w" sizes="(max-width: 326px) 100vw, 326px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h3><em>JavaScript:</em></h3> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> <span class="token function-variable function">App</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> <span class="token punctuation">[</span>count<span class="token punctuation">,</span> setCount<span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token function">useState</span><span class="token punctuation">(</span><span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> <span class="token function-variable function">increment</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">setCount</span><span class="token punctuation">(</span>count <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> <span class="token function-variable function">decrement</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">setCount</span><span class="token punctuation">(</span>count <span class="token operator">-</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token punctuation">(</span> <span class="token operator">&lt;</span>div<span class="token operator">></span> <span class="token operator">&lt;</span>button onClick<span class="token operator">=</span><span class="token punctuation">{</span>decrement<span class="token punctuation">}</span><span class="token operator">></span><span class="token operator">-</span><span class="token operator">&lt;</span><span class="token operator">/</span>button<span class="token operator">></span> <span class="token punctuation">{</span>count<span class="token punctuation">}</span> <span class="token operator">&lt;</span>button onClick<span class="token operator">=</span><span class="token punctuation">{</span>increment<span class="token punctuation">}</span><span class="token operator">></span><span class="token operator">+</span><span class="token operator">&lt;</span><span class="token operator">/</span>button<span class="token operator">></span> <span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre></div> <h3><em>Elm:</em></h3> <div class="gatsby-highlight" data-language="haskell"><pre class="language-haskell"><code class="language-haskell"><span class="token keyword">type</span> <span class="token constant">Msg</span> <span class="token operator">=</span> <span class="token constant">Increment</span> <span class="token operator">|</span> <span class="token constant">Decrement</span> <span class="token builtin">init</span> <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token hvariable">count</span> <span class="token operator">=</span> <span class="token number">0</span> <span class="token punctuation">}</span> <span class="token hvariable">update</span> <span class="token hvariable">msg</span> <span class="token hvariable">model</span> <span class="token operator">=</span> <span class="token keyword">case</span> <span class="token hvariable">msg</span> <span class="token keyword">of</span> <span class="token constant">Increment</span> <span class="token operator">-></span> <span class="token punctuation">{</span> <span class="token hvariable">model</span> <span class="token operator">|</span> <span class="token hvariable">count</span> <span class="token operator">=</span> <span class="token hvariable">model</span><span class="token punctuation">.</span><span class="token hvariable">count</span> <span class="token operator">+</span> <span class="token number">1</span> <span class="token punctuation">}</span> <span class="token constant">Decrement</span> <span class="token operator">-></span> <span class="token punctuation">{</span> <span class="token hvariable">model</span> <span class="token operator">|</span> <span class="token hvariable">count</span> <span class="token operator">=</span> <span class="token hvariable">model</span><span class="token punctuation">.</span><span class="token hvariable">count</span> <span class="token operator">-</span> <span class="token number">1</span> <span class="token punctuation">}</span> <span class="token hvariable">view</span> <span class="token hvariable">model</span> <span class="token operator">=</span> <span class="token builtin">div</span> <span class="token punctuation">[</span><span class="token punctuation">]</span> <span class="token punctuation">[</span> <span class="token hvariable">button</span> <span class="token punctuation">[</span> <span class="token hvariable">onClick</span> <span class="token constant">Decrement</span> <span class="token punctuation">]</span> <span class="token punctuation">[</span> <span class="token hvariable">text</span> <span class="token string">"-"</span> <span class="token punctuation">]</span> <span class="token punctuation">,</span> <span class="token hvariable">text</span> <span class="token punctuation">(</span><span class="token constant">String</span><span class="token punctuation">.</span><span class="token builtin">fromInt</span> <span class="token hvariable">model</span><span class="token punctuation">.</span><span class="token hvariable">count</span><span class="token punctuation">)</span> <span class="token punctuation">,</span> <span class="token hvariable">button</span> <span class="token punctuation">[</span> <span class="token hvariable">onClick</span> <span class="token constant">Increment</span> <span class="token punctuation">]</span> <span class="token punctuation">[</span> <span class="token hvariable">text</span> <span class="token string">"+"</span> <span class="token punctuation">]</span> <span class="token punctuation">]</span></code></pre></div> <p>As you can see, the two programs are similar in many ways, and quite different in others. If you're struggling with the somewhat alien ML-style syntax, check out <a href="https://elm-lang.org/docs/from-JavaScript">this quick syntax guide</a>.</p> <p>To start, we define our <em><code>Msg</code></em> type to be either an <em><code>Increment</code></em> or <em><code>Decrement</code></em> value - more on this later.</p> <p>After that, the Elm program is split across 3 self-contained function definitions, <strong><code>init</code></strong>, <strong><code>update</code></strong> and <strong><code>view</code></strong>. This contrasts with React, where everything is contained within one function.</p> <p>The <strong><code>init</code></strong> function is much like the 2nd line of our React app: it defines the initial state of the <em>Model</em>, essentially the datatype for the entire application's state. Unlike when we use React's Hooks, we are encouraged to store all the state for our entire application in one data structure, for reasons discussed later.</p> <p>The <strong><code>update</code></strong> function then takes two arguments, the <em>Model</em> and a <em>Message</em>. It then decides how to change the model based on the message received, returning a new copy of the model with the count changed. You can read the code <code>{ model | count = model.count + 1 }</code> as <em>'a copy of model where its count is one greater'</em>. Elm calls this function for us whenever a new message is sent, for example on a button press.</p> <p>Finally, the <strong><code>view</code></strong> function then takes one argument, the <em>Model</em>, and renders it as HTML. Other than the style, this is mostly the same as the return statement in the React function - except that in Elm our buttons send <em>Messages</em> rather than call lambdas. Elm calls <strong><code>view</code></strong> to re-render the page whenever the <em>Model</em> changes.</p> <h2>What benefits does this style provide?</h2> <p>Most of these differences are a result of the fact that Elm is a <em>purely functional</em> language, meaning we aren't allowed to perform any <em>'side effects'</em> as we would in React with Hooks. Functional purity gives us important guarantees about our codebase - we can be sure that, given the same inputs, we will always get the same output. In a practical sense, this makes the code easier to read - you know the only values that will change in a function are its parameters, so you don't have to waste time reading the whole surrounding context. This is one reason why Elm files can be longer than JavaScript ones without compromising maintainability, as the smallest unit you have to comprehend is a function, not an entire module.</p> <p>However, separating out the code into <strong><code>init</code></strong>, <strong><code>update</code></strong> and <strong><code>view</code></strong> has other benefits - the separation between code describing the application logic and how it looks is now enforced by the language.</p> <p><img src="/9aa3b25cf36113089e2c99d730e09795/buttons.svg" alt="diagram of the elm architecture"></p> <p>The guide refers to this structure as <em>'The Elm Architecture'</em> (see diagram above). The Elm runtime manages our application for us, calling <strong><code>view</code></strong> whenever the model changes and <strong><code>update</code></strong> with every message received. The interactions here will be familiar to users of Redux, which was inspired by Elm, where a Redux <em>'Action'</em> roughly approximates an Elm <em>Message</em>, and <strong><code>update</code></strong> is akin to a <em>'Reducer'</em>.</p> <p>Having a centralised <em>Msg</em> and <em>Model</em> type can boost productivity, as changing these types in a large module will cause the compiler to alert us to all the places that subsequently need updating, freeing us from the task of searching through the codebase manually. Centralising the state into a single <em>Model</em> also has architectural benefits. It allows us to <a href="https://www.youtube.com/watch?v=IcgmSRJHu_8">make illegal states un-representable</a>, and forces us to thoroughly describe all the dynamic aspects of our webpage. Modelling the state of our application as a data structure in this way is a powerful tool, that helps us consider what really lies at the core of our application.</p> <p>A stricter type system can seem like a limitation, as an example, where React lets us simply write <strong><code>{count}</code></strong>, Elm makes us write <strong><code>text (String.fromInt model.count)</code></strong>, converting the <strong><code>Int</code></strong> to a <strong><code>String</code></strong> to <strong><code>Html</code></strong>. However, as TypeScript evangelists will tell you, the advantages of type safety often outweigh the additional work, as it increases the predictability of the code and, in Elm's case, eliminates the possibility of a runtime error.</p> <h2>Where are these famous types?</h2> <p>In Elm, types can usually be inferred from the code you write, so you almost never need to write type annotations to reap the rewards of type safety. However, they are often considered good practice as they can make the code more readable and help the compiler show better error messages. So, for the sake of interest, here is the above program with type annotations:</p> <div class="gatsby-highlight" data-language="haskell"><pre class="language-haskell"><code class="language-haskell"><span class="token keyword">type</span> <span class="token constant">Msg</span> <span class="token operator">=</span> <span class="token constant">Increment</span> <span class="token operator">|</span> <span class="token constant">Decrement</span> <span class="token keyword">type</span> <span class="token hvariable">alias</span> <span class="token constant">Model</span> <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token hvariable">count</span> <span class="token operator">:</span> <span class="token constant">Int</span> <span class="token punctuation">}</span> <span class="token builtin">init</span> <span class="token operator">:</span> <span class="token constant">Model</span> <span class="token builtin">init</span> <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token hvariable">count</span> <span class="token operator">=</span> <span class="token number">0</span> <span class="token punctuation">}</span> <span class="token hvariable">update</span> <span class="token operator">:</span> <span class="token constant">Msg</span> <span class="token operator">-></span> <span class="token constant">Model</span> <span class="token operator">-></span> <span class="token constant">Model</span> <span class="token hvariable">update</span> <span class="token hvariable">msg</span> <span class="token hvariable">model</span> <span class="token operator">=</span> <span class="token keyword">case</span> <span class="token hvariable">msg</span> <span class="token keyword">of</span> <span class="token constant">Increment</span> <span class="token operator">-></span> <span class="token punctuation">{</span> <span class="token hvariable">model</span> <span class="token operator">|</span> <span class="token hvariable">count</span> <span class="token operator">=</span> <span class="token hvariable">model</span><span class="token punctuation">.</span><span class="token hvariable">count</span> <span class="token operator">+</span> <span class="token number">1</span> <span class="token punctuation">}</span> <span class="token constant">Decrement</span> <span class="token operator">-></span> <span class="token punctuation">{</span> <span class="token hvariable">model</span> <span class="token operator">|</span> <span class="token hvariable">count</span> <span class="token operator">=</span> <span class="token hvariable">model</span><span class="token punctuation">.</span><span class="token hvariable">count</span> <span class="token operator">-</span> <span class="token number">1</span> <span class="token punctuation">}</span> <span class="token hvariable">view</span> <span class="token operator">:</span> <span class="token constant">Model</span> <span class="token operator">-></span> <span class="token constant">Html</span> <span class="token constant">Msg</span> <span class="token hvariable">view</span> <span class="token hvariable">model</span> <span class="token operator">=</span> <span class="token builtin">div</span> <span class="token punctuation">[</span><span class="token punctuation">]</span> <span class="token punctuation">[</span> <span class="token hvariable">button</span> <span class="token punctuation">[</span> <span class="token hvariable">onClick</span> <span class="token constant">Decrement</span> <span class="token punctuation">]</span> <span class="token punctuation">[</span> <span class="token hvariable">text</span> <span class="token string">"-"</span> <span class="token punctuation">]</span> <span class="token punctuation">,</span> <span class="token hvariable">text</span> <span class="token punctuation">(</span><span class="token constant">String</span><span class="token punctuation">.</span><span class="token builtin">fromInt</span> <span class="token hvariable">model</span><span class="token punctuation">.</span><span class="token hvariable">count</span><span class="token punctuation">)</span> <span class="token punctuation">,</span> <span class="token hvariable">button</span> <span class="token punctuation">[</span> <span class="token hvariable">onClick</span> <span class="token constant">Increment</span> <span class="token punctuation">]</span> <span class="token punctuation">[</span> <span class="token hvariable">text</span> <span class="token string">"+"</span> <span class="token punctuation">]</span> <span class="token punctuation">]</span></code></pre></div> <p>First, we give the <em>Model</em> a formal type. We use the syntax <strong><code>type alias</code></strong> here, because we're not introducing a new set of values (as with Msg), simply a shorthand for <strong><code>{ count : Int }</code></strong>, a record with an integer field called <em>'count'</em>. We use this new type to annotate the declaration of <strong><code>init</code></strong>, saying it is just a constant of type <strong><code>Model</code></strong>.</p> <p>Next, we give <strong><code>update</code></strong> a type annotation, <strong><code>Msg -> Model -> Model</code></strong>. This is essentially saying <em>'update is a function which takes a Message and a Model, and returns another Model'</em>. This is somewhat obfuscated by the fact Elm functions are <em>'curried'</em> by default, making the actual meaning more like <em>'update is a function which takes a Message, and returns a function which takes a Model and returns a Model'</em>.</p> <p>Finally, we annotate <strong><code>view</code></strong> with the type <strong><code>Model -> Html Msg</code></strong>. This states that the <strong><code>view</code></strong> function takes a model and returns some HTML, specifically HTML with our custom <em>Messages</em> embedded within it. This is a form of parametrised type: the TypeScript equivalent would be <strong><code>Html&#x3C;Msg></code></strong>.</p> <h2>What next?</h2> <p>If you're interested in experimenting with Elm, you can neatly integrate Elm components into your personal React projects using <a href="https://www.npmjs.com/package/@elm-react/component">@elm-react/component</a>.</p> <p>For those interested in learning more about the language and its influence on frontend development, I'd recommend one of my favourite talks, <em><a href="https://www.youtube.com/watch?v=XpDsk374LDE">'The life of a file'</a></em> by Evan Czaplicki, Elm's creator.</p> <p>Thanks to the compiler's helpful and friendly error messages, the language almost teaches itself (after a brief skim of <a href="https://guide.elm-lang.org/">the guide</a>) - so if you're a <em>'learn-by-doing'</em> type of developer I recommend <a href="https://github.com/wking-io/elm-live#readme">elm-live</a> to re-run the compiler every time you change your file. It's an interesting, even somewhat calming way of programming, and while the language may not have seen mainstream success yet, I think it can continue to show us a glimpse of the future of web development.</p><![CDATA[Migrating from React Redux to React Query]]>/2021/10/migrating-from-react-redux-to-react-query//2021/10/migrating-from-react-redux-to-react-query/Tue, 19 Oct 2021 00:00:00 GMT<p>React Redux is an extremely popular package, receiving about 4.5 million weekly downloads. Another popular package, Redux Saga, is a Redux side effect manager — it alone receives over 900k weekly downloads. The combination of React Redux with either Redux Saga or Redux Thunk is one of the most common additions to React projects. </p> <p>However, over the last year or so, React Query has been gaining popularity quickly as an alternative to React Redux. Last year, React Query had only about 120k weekly downloads, but as of October 2021, React Query downloads are nearly six times more frequent, at nearly 700k downloads per week. </p> <p>React Query has some distinct advantages over React Redux for storing server state. </p> <p>A major advantage of React Query is that it is far simpler to write than React Redux. In React Redux, an operation to optimistically update a field requires three actions (request, success, and failure), three reducers (request, success, and failure), one middleware, and a selector to access the data. In React Query however, the equivalent setup is significantly simpler as it only requires one mutation (which consists of four functions) and one query. </p> <p>Another advantage of React Query is that it works more smoothly with navigation tools built into many IDEs. With React Query, you can use “Go To Definition” to more easily reach your mutation whereas with Redux Saga, you must search the entire codebase for uses of the action. </p> <p>On any existing project, making the switch to use a new package can be a daunting task. Today, to help make this task more manageable, we will be looking at an example where an application using React Redux with Redux Saga will be migrated to React Query.</p> <h2>Our Initial Project</h2> <p>To start, let's take a look at a basic optimistic Redux setup with three reducers, three actions, a selector, and a saga. </p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 375px; " > <a class="gatsby-resp-image-link" href="/static/8f2695d864dbcb6b5cb05f689237b7a3/5ff7e/site-no-name.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 20%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAECAYAAACOXx+WAAAACXBIWXMAAAsSAAALEgHS3X78AAAAp0lEQVQY012Q2wqEMAxE+//fpr4VhYJIvRQVERUfVOosM+DibmBomstJWnPfN2KMEq3rOvR9j2EY4L3XOY4j6rpWLoSgGP1t29RDBkUz+LN1XQVkY1VVXwiBD5QDCN33/aeXULMsC6y1Ki7LEs45MHaep3Qch/Tcr+vCNE3I8xxJkiDLMqRpiqIolDMsZgHXn+dZPie/YW8xztq2bfUCbt80jbbmt30AqzYv9JZjUgQAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="example site without name entered" title="example site without name entered" src="/static/8f2695d864dbcb6b5cb05f689237b7a3/5ff7e/site-no-name.png" srcset="/static/8f2695d864dbcb6b5cb05f689237b7a3/1d79a/site-no-name.png 185w, /static/8f2695d864dbcb6b5cb05f689237b7a3/1efb2/site-no-name.png 370w, /static/8f2695d864dbcb6b5cb05f689237b7a3/5ff7e/site-no-name.png 375w" sizes="(max-width: 375px) 100vw, 375px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span> <span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 375px; " > <a class="gatsby-resp-image-link" href="/static/a283de651460ed24f97209aaa99942f4/5ff7e/site-with-name.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 20%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAECAYAAACOXx+WAAAACXBIWXMAAAsSAAALEgHS3X78AAAArUlEQVQY03VQywqEMBDr/3+aIB5EUPGB1ieiIl58Z0nAy7IbGGbaSTOZmud5cN833kxs2wZrLYZhQF3XaJpGdZqmKMsS4ziK/x2EwQ+w2fc9pmmSQFVVOhdFgTzPdf/vnVmWBUEQyFGWZYjjWK6iKEKSJHJ7HAf2fVe+rksOfd+H4zjwPA+u6yIMQ5znCUMiCeu6Yp5nTWfddZ3WpCA5b1CUfQ6l47ZttQG5/LIPmo4wNlp0csAAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="example site with name entered" title="example site with name entered" src="/static/a283de651460ed24f97209aaa99942f4/5ff7e/site-with-name.png" srcset="/static/a283de651460ed24f97209aaa99942f4/1d79a/site-with-name.png 185w, /static/a283de651460ed24f97209aaa99942f4/1efb2/site-with-name.png 370w, /static/a283de651460ed24f97209aaa99942f4/5ff7e/site-with-name.png 375w" sizes="(max-width: 375px) 100vw, 375px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>In our example, we have a React website where a user can input their name and their name is displayed on the page. A selector is used on the React page to display the user’s inputted name. When a user inputs their name, a request action is dispatched that causes a reducer and a saga to run. The reducer optimistically updates the store with the new value of the name. Meanwhile, the saga makes a call to the backend to save the user’s name. </p> <p>If the backend returns an HTTP success, the saga dispatches a success action. The success action causes a reducer to run which updates the value of the name in the store. </p> <p>If the backend returns an HTTP failure, the saga dispatches a failure action. The failure action causes a reducer to run which changes the value in the store back to its value before the optimistic update.</p> <p><strong><em>NameComponent.tsx</em></strong></p> <div class="gatsby-highlight" data-language="ts"><pre class="language-ts"><code class="language-ts"><span class="token keyword">import</span> React<span class="token punctuation">,</span> <span class="token punctuation">{</span> ChangeEvent<span class="token punctuation">,</span> FormEvent<span class="token punctuation">,</span> FunctionComponent<span class="token punctuation">,</span> useCallback<span class="token punctuation">,</span> useEffect<span class="token punctuation">,</span> useState<span class="token punctuation">,</span> <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"react"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> useDispatch<span class="token punctuation">,</span> useSelector <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"react-redux"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> selectName<span class="token punctuation">,</span> updateNameRequest <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"../redux/slice"</span><span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token keyword">const</span> NameComponent<span class="token operator">:</span> <span class="token function-variable function">FunctionComponent</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> name <span class="token operator">=</span> <span class="token function">useSelector</span><span class="token punctuation">(</span>selectName<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> <span class="token punctuation">[</span>localName<span class="token punctuation">,</span> setLocalName<span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token generic-function"><span class="token function">useState</span><span class="token generic class-name"><span class="token operator">&lt;</span><span class="token builtin">string</span><span class="token operator">></span></span></span><span class="token punctuation">(</span><span class="token string">""</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> dispatch <span class="token operator">=</span> <span class="token function">useDispatch</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">useEffect</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>name<span class="token punctuation">)</span> <span class="token function">setLocalName</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">[</span>name<span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> handleChange <span class="token operator">=</span> <span class="token function">useCallback</span><span class="token punctuation">(</span><span class="token punctuation">(</span>event<span class="token operator">:</span> ChangeEvent<span class="token operator">&lt;</span>HTMLInputElement<span class="token operator">></span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token function">setLocalName</span><span class="token punctuation">(</span>event<span class="token punctuation">.</span>target<span class="token punctuation">.</span>value<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> handleSubmit <span class="token operator">=</span> <span class="token function">useCallback</span><span class="token punctuation">(</span> <span class="token punctuation">(</span>event<span class="token operator">:</span> FormEvent<span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token function">dispatch</span><span class="token punctuation">(</span><span class="token function">updateNameRequest</span><span class="token punctuation">(</span><span class="token punctuation">{</span> name<span class="token operator">:</span> localName <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> event<span class="token punctuation">.</span><span class="token function">preventDefault</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">[</span>dispatch<span class="token punctuation">,</span> localName<span class="token punctuation">]</span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token punctuation">(</span> <span class="token operator">&lt;</span>div<span class="token operator">></span> <span class="token operator">&lt;</span>p<span class="token operator">></span><span class="token punctuation">{</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Your name is </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>name <span class="token operator">??</span> <span class="token string">"unknown to me"</span><span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">.</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>p<span class="token operator">></span> <span class="token operator">&lt;</span>form onSubmit<span class="token operator">=</span><span class="token punctuation">{</span>handleSubmit<span class="token punctuation">}</span><span class="token operator">></span> <span class="token operator">&lt;</span>label<span class="token operator">></span> <span class="token punctuation">{</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Enter name: </span><span class="token template-punctuation string">`</span></span><span class="token punctuation">}</span> <span class="token operator">&lt;</span>input <span class="token keyword">type</span><span class="token operator">=</span><span class="token string">"text"</span> onChange<span class="token operator">=</span><span class="token punctuation">{</span>handleChange<span class="token punctuation">}</span> <span class="token operator">/</span><span class="token operator">></span> <span class="token operator">&lt;</span><span class="token operator">/</span>label<span class="token operator">></span> <span class="token operator">&lt;</span>input <span class="token keyword">type</span><span class="token operator">=</span><span class="token string">"submit"</span> value<span class="token operator">=</span><span class="token string">"Update Name"</span> <span class="token operator">/</span><span class="token operator">></span> <span class="token operator">&lt;</span><span class="token operator">/</span>form<span class="token operator">></span> <span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre></div> <p><strong><em>slice.ts</em></strong></p> <div class="gatsby-highlight" data-language="ts"><pre class="language-ts"><code class="language-ts"><span class="token keyword">import</span> <span class="token punctuation">{</span> createSlice<span class="token punctuation">,</span> PayloadAction <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"@reduxjs/toolkit"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> RootState <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">".."</span><span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token keyword">interface</span> <span class="token class-name">NameState</span> <span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token builtin">string</span> <span class="token operator">|</span> <span class="token keyword">null</span><span class="token punctuation">;</span> error<span class="token operator">:</span> <span class="token builtin">string</span> <span class="token operator">|</span> <span class="token keyword">null</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token keyword">export</span> <span class="token keyword">const</span> initialState<span class="token operator">:</span> NameState <span class="token operator">=</span> <span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token keyword">null</span><span class="token punctuation">,</span> error<span class="token operator">:</span> <span class="token keyword">null</span><span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token keyword">const</span> nameSlice <span class="token operator">=</span> <span class="token function">createSlice</span><span class="token punctuation">(</span><span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token string">"name"</span><span class="token punctuation">,</span> initialState<span class="token punctuation">,</span> reducers<span class="token operator">:</span> <span class="token punctuation">{</span> <span class="token function-variable function">updateNameRequest</span><span class="token operator">:</span> <span class="token punctuation">(</span>state<span class="token punctuation">,</span> action<span class="token operator">:</span> PayloadAction<span class="token operator">&lt;</span><span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token builtin">string</span> <span class="token punctuation">}</span><span class="token operator">></span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token operator">...</span>state<span class="token punctuation">,</span> <span class="token operator">...</span>action<span class="token punctuation">.</span>payload<span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token function-variable function">updateNameSuccess</span><span class="token operator">:</span> <span class="token punctuation">(</span>state<span class="token punctuation">,</span> action<span class="token operator">:</span> PayloadAction<span class="token operator">&lt;</span><span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token builtin">string</span> <span class="token punctuation">}</span><span class="token operator">></span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token operator">...</span>state<span class="token punctuation">,</span> <span class="token operator">...</span>action<span class="token punctuation">.</span>payload<span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token function-variable function">updateNameFailure</span><span class="token operator">:</span> <span class="token punctuation">(</span> state<span class="token punctuation">,</span> action<span class="token operator">:</span> PayloadAction<span class="token operator">&lt;</span><span class="token punctuation">{</span> error<span class="token operator">:</span> <span class="token builtin">string</span><span class="token punctuation">;</span> name<span class="token operator">:</span> <span class="token builtin">string</span> <span class="token punctuation">}</span><span class="token operator">></span> <span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token operator">...</span>state<span class="token punctuation">,</span> <span class="token operator">...</span>action<span class="token punctuation">.</span>payload<span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">selectName</span> <span class="token operator">=</span> <span class="token punctuation">(</span>state<span class="token operator">:</span> RootState<span class="token punctuation">)</span> <span class="token operator">=></span> state<span class="token punctuation">.</span>name<span class="token punctuation">;</span> <span class="token keyword">const</span> <span class="token punctuation">{</span> actions<span class="token punctuation">,</span> reducer <span class="token punctuation">}</span> <span class="token operator">=</span> nameSlice<span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token punctuation">{</span> updateNameRequest<span class="token punctuation">,</span> updateNameSuccess<span class="token punctuation">,</span> updateNameFailure <span class="token punctuation">}</span> <span class="token operator">=</span> actions<span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token keyword">type</span> <span class="token class-name">NameAction</span> <span class="token operator">=</span> <span class="token operator">|</span> <span class="token keyword">typeof</span> updateNameRequest <span class="token operator">|</span> <span class="token keyword">typeof</span> updateNameSuccess <span class="token operator">|</span> <span class="token keyword">typeof</span> updateNameFailure<span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token punctuation">{</span> reducer <span class="token keyword">as</span> nameReducer <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre></div> <p><strong><em>sagas.ts</em></strong></p> <div class="gatsby-highlight" data-language="ts"><pre class="language-ts"><code class="language-ts"><span class="token keyword">import</span> <span class="token punctuation">{</span> ActionType <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"typesafe-actions"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> call<span class="token punctuation">,</span> put<span class="token punctuation">,</span> select<span class="token punctuation">,</span> takeEvery <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"redux-saga/effects"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> selectName<span class="token punctuation">,</span> updateNameFailure<span class="token punctuation">,</span> updateNameRequest<span class="token punctuation">,</span> updateNameSuccess<span class="token punctuation">,</span> <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"./slice"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> updateName <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"../api/api"</span><span class="token punctuation">;</span> <span class="token keyword">function</span><span class="token operator">*</span> <span class="token function">updateNameSaga</span><span class="token punctuation">(</span>action<span class="token operator">:</span> ActionType<span class="token operator">&lt;</span><span class="token keyword">typeof</span> updateNameRequest<span class="token operator">></span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">const</span> <span class="token punctuation">{</span> name <span class="token punctuation">}</span> <span class="token operator">=</span> action<span class="token punctuation">.</span>payload<span class="token punctuation">;</span> <span class="token keyword">const</span> oldName<span class="token operator">:</span> <span class="token builtin">string</span> <span class="token operator">=</span> <span class="token keyword">yield</span> <span class="token function">select</span><span class="token punctuation">(</span>selectName<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">try</span> <span class="token punctuation">{</span> <span class="token keyword">yield</span> <span class="token function">call</span><span class="token punctuation">(</span>updateName<span class="token punctuation">,</span> name<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">yield</span> <span class="token function">put</span><span class="token punctuation">(</span><span class="token function">updateNameSuccess</span><span class="token punctuation">(</span>action<span class="token punctuation">.</span>payload<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span>error<span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">yield</span> <span class="token function">put</span><span class="token punctuation">(</span> <span class="token function">updateNameFailure</span><span class="token punctuation">(</span><span class="token punctuation">{</span> error<span class="token operator">:</span> <span class="token punctuation">(</span>error <span class="token keyword">as</span> Error<span class="token punctuation">)</span><span class="token punctuation">.</span>message<span class="token punctuation">,</span> name<span class="token operator">:</span> oldName <span class="token punctuation">}</span><span class="token punctuation">)</span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span> <span class="token keyword">export</span> <span class="token keyword">function</span><span class="token operator">*</span> <span class="token function">nameSagas</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">yield</span> <span class="token function">takeEvery</span><span class="token punctuation">(</span>updateNameRequest<span class="token punctuation">,</span> updateNameSaga<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></code></pre></div> <p><strong><em>store.ts</em></strong></p> <div class="gatsby-highlight" data-language="ts"><pre class="language-ts"><code class="language-ts"><span class="token keyword">import</span> <span class="token punctuation">{</span> applyMiddleware<span class="token punctuation">,</span> createStore <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"@reduxjs/toolkit"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> createSagaMiddleware <span class="token keyword">from</span> <span class="token string">"redux-saga"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> nameSagas <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"../redux/sagas"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> nameReducer <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"../redux/slice"</span><span class="token punctuation">;</span> <span class="token keyword">const</span> sagaMiddleware <span class="token operator">=</span> <span class="token function">createSagaMiddleware</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token keyword">default</span> <span class="token keyword">function</span> <span class="token function">configureStore</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">const</span> store <span class="token operator">=</span> <span class="token function">createStore</span><span class="token punctuation">(</span>nameReducer<span class="token punctuation">,</span> <span class="token function">applyMiddleware</span><span class="token punctuation">(</span>sagaMiddleware<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> sagaMiddleware<span class="token punctuation">.</span><span class="token function">run</span><span class="token punctuation">(</span>nameSagas<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token punctuation">{</span> store <span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></code></pre></div> <h2>Converting the Redux Store to React Query</h2> <p>The first step of our conversion is to create a query to replace the selector and the store. Below is a custom React hook that contains a useQuery to store the user’s name. In this example, getName is a call to the backend to get the name stored in the database.</p> <div class="gatsby-highlight" data-language="ts"><pre class="language-ts"><code class="language-ts"><span class="token keyword">import</span> <span class="token punctuation">{</span> getName <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"../api/api"</span><span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getNameQuery</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">(</span><span class="token punctuation">{</span> queryKey<span class="token operator">:</span> <span class="token punctuation">[</span><span class="token string">"name"</span><span class="token punctuation">]</span><span class="token punctuation">,</span> queryFn<span class="token operator">:</span> getName<span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre></div> <h2>Converting the Saga to a Mutation</h2> <p>The next step of our conversion is to create an optimistic mutation to replace the saga, reducers, and Redux actions. Below is an optimistic mutation in a custom React hook that optimistically updates the query and invalidates the query after the update.</p> <div class="gatsby-highlight" data-language="ts"><pre class="language-ts"><code class="language-ts"><span class="token keyword">import</span> <span class="token punctuation">{</span> useMutation<span class="token punctuation">,</span> useQueryClient <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"react-query"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> updateName <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"../api/api"</span><span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">useEditName</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> qc <span class="token operator">=</span> <span class="token function">useQueryClient</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token generic-function"><span class="token function">useMutation</span><span class="token generic class-name"><span class="token operator">&lt;</span><span class="token keyword">void</span><span class="token punctuation">,</span> <span class="token builtin">unknown</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> newName<span class="token operator">:</span> <span class="token builtin">string</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token builtin">string</span><span class="token operator">></span></span></span><span class="token punctuation">(</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">{</span> newName <span class="token punctuation">}</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">await</span> <span class="token function">updateName</span><span class="token punctuation">(</span>newName<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> <span class="token function-variable function">onMutate</span><span class="token operator">:</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">{</span> newName <span class="token punctuation">}</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">await</span> qc<span class="token punctuation">.</span><span class="token function">cancelQueries</span><span class="token punctuation">(</span><span class="token string">"name"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> currentName<span class="token operator">:</span> <span class="token builtin">string</span> <span class="token operator">|</span> <span class="token keyword">undefined</span> <span class="token operator">=</span> qc<span class="token punctuation">.</span><span class="token function">getQueryData</span><span class="token punctuation">(</span><span class="token string">"name"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> qc<span class="token punctuation">.</span><span class="token function">setQueryData</span><span class="token punctuation">(</span><span class="token string">"name"</span><span class="token punctuation">,</span> newName<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> currentName<span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token function-variable function">onError</span><span class="token operator">:</span> <span class="token punctuation">(</span>_error<span class="token punctuation">,</span> _variables<span class="token punctuation">,</span> context<span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>context<span class="token punctuation">)</span> qc<span class="token punctuation">.</span><span class="token function">setQueryData</span><span class="token punctuation">(</span><span class="token string">"name"</span><span class="token punctuation">,</span> context<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token function-variable function">onSettled</span><span class="token operator">:</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> qc<span class="token punctuation">.</span><span class="token function">invalidateQueries</span><span class="token punctuation">(</span><span class="token string">"name"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">}</span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre></div> <p>The mutation function in this case calls the backend with updateName to change the name stored in the database.</p> <p>Before the mutation function, Redux Query executes <strong>onMutate</strong>. In this case, we cancel the query that we are updating, get the current value for the query, and update the value for the query with the new value. We then return the old value for the query so we can revert if our optimism does not pay off. </p> <p>If the mutation function throws an error, Redux Query executes <strong>onError</strong>. All we need to do is set the query to the context parameter if the backend call fails. </p> <p>Lastly, Redux Query executes <strong>onSettled</strong>, which simply invalidates the query so Redux Query knows that the data is stale. This forces Redux Query to get the name fresh from the backend.</p> <h2>Final Project</h2> <p>The new query and mutation can be used in the following way in the React component. In the end, we will have a React Query setup with an optimistic mutation. Just two hooks replace the original React Redux and Redux Saga setup. </p> <p><strong><em>name.ts</em></strong></p> <div class="gatsby-highlight" data-language="ts"><pre class="language-ts"><code class="language-ts"><span class="token keyword">import</span> <span class="token punctuation">{</span> useMutation<span class="token punctuation">,</span> useQueryClient <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"react-query"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> getName<span class="token punctuation">,</span> updateName <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"../api/api"</span><span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getNameQuery</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">(</span><span class="token punctuation">{</span> queryKey<span class="token operator">:</span> <span class="token punctuation">[</span><span class="token string">"name"</span><span class="token punctuation">]</span><span class="token punctuation">,</span> queryFn<span class="token operator">:</span> getName<span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">useEditName</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> qc <span class="token operator">=</span> <span class="token function">useQueryClient</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token generic-function"><span class="token function">useMutation</span><span class="token generic class-name"><span class="token operator">&lt;</span><span class="token keyword">void</span><span class="token punctuation">,</span> <span class="token builtin">unknown</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> newName<span class="token operator">:</span> <span class="token builtin">string</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token builtin">string</span><span class="token operator">></span></span></span><span class="token punctuation">(</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">{</span> newName <span class="token punctuation">}</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">await</span> <span class="token function">updateName</span><span class="token punctuation">(</span>newName<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> <span class="token function-variable function">onMutate</span><span class="token operator">:</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">{</span> newName <span class="token punctuation">}</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">await</span> qc<span class="token punctuation">.</span><span class="token function">cancelQueries</span><span class="token punctuation">(</span><span class="token string">"name"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> currentName<span class="token operator">:</span> <span class="token builtin">string</span> <span class="token operator">|</span> <span class="token keyword">undefined</span> <span class="token operator">=</span> qc<span class="token punctuation">.</span><span class="token function">getQueryData</span><span class="token punctuation">(</span><span class="token string">"name"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> qc<span class="token punctuation">.</span><span class="token function">setQueryData</span><span class="token punctuation">(</span><span class="token string">"name"</span><span class="token punctuation">,</span> newName<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> currentName<span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token function-variable function">onError</span><span class="token operator">:</span> <span class="token punctuation">(</span>_error<span class="token punctuation">,</span> _variables<span class="token punctuation">,</span> context<span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>context<span class="token punctuation">)</span> qc<span class="token punctuation">.</span><span class="token function">setQueryData</span><span class="token punctuation">(</span><span class="token string">"name"</span><span class="token punctuation">,</span> context<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token function-variable function">onSettled</span><span class="token operator">:</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> qc<span class="token punctuation">.</span><span class="token function">invalidateQueries</span><span class="token punctuation">(</span><span class="token string">"name"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">}</span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre></div> <p><strong><em>NameComponent.tsx</em></strong></p> <div class="gatsby-highlight" data-language="ts"><pre class="language-ts"><code class="language-ts"><span class="token keyword">import</span> React<span class="token punctuation">,</span> <span class="token punctuation">{</span> ChangeEvent<span class="token punctuation">,</span> FormEvent<span class="token punctuation">,</span> FunctionComponent<span class="token punctuation">,</span> useCallback<span class="token punctuation">,</span> useEffect<span class="token punctuation">,</span> useState<span class="token punctuation">,</span> <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"react"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> useQuery <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"react-query"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> getNameQuery<span class="token punctuation">,</span> useEditName <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"../dataLayer/name"</span><span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token keyword">const</span> NameComponent<span class="token operator">:</span> <span class="token function-variable function">FunctionComponent</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">const</span> editNameMutation <span class="token operator">=</span> <span class="token function">useEditName</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> <span class="token punctuation">{</span> data<span class="token operator">:</span> name <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">useQuery</span><span class="token punctuation">(</span><span class="token function">getNameQuery</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> <span class="token punctuation">[</span>localName<span class="token punctuation">,</span> setLocalName<span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token generic-function"><span class="token function">useState</span><span class="token generic class-name"><span class="token operator">&lt;</span><span class="token builtin">string</span><span class="token operator">></span></span></span><span class="token punctuation">(</span><span class="token string">""</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">useEffect</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>name<span class="token punctuation">)</span> <span class="token function">setLocalName</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">[</span>name<span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> handleChange <span class="token operator">=</span> <span class="token function">useCallback</span><span class="token punctuation">(</span><span class="token punctuation">(</span>event<span class="token operator">:</span> ChangeEvent<span class="token operator">&lt;</span>HTMLInputElement<span class="token operator">></span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token function">setLocalName</span><span class="token punctuation">(</span>event<span class="token punctuation">.</span>target<span class="token punctuation">.</span>value<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">const</span> handleSubmit <span class="token operator">=</span> <span class="token function">useCallback</span><span class="token punctuation">(</span> <span class="token punctuation">(</span>event<span class="token operator">:</span> FormEvent<span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> editNameMutation<span class="token punctuation">.</span><span class="token function">mutate</span><span class="token punctuation">(</span><span class="token punctuation">{</span> newName<span class="token operator">:</span> localName <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span> event<span class="token punctuation">.</span><span class="token function">preventDefault</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">[</span>editNameMutation<span class="token punctuation">,</span> localName<span class="token punctuation">]</span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token punctuation">(</span> <span class="token operator">&lt;</span>div<span class="token operator">></span> <span class="token operator">&lt;</span>p<span class="token operator">></span><span class="token punctuation">{</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Your name is </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>name <span class="token operator">??</span> <span class="token string">"unknown to me"</span><span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">.</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>p<span class="token operator">></span> <span class="token operator">&lt;</span>form onSubmit<span class="token operator">=</span><span class="token punctuation">{</span>handleSubmit<span class="token punctuation">}</span><span class="token operator">></span> <span class="token operator">&lt;</span>label<span class="token operator">></span> <span class="token punctuation">{</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Enter name: </span><span class="token template-punctuation string">`</span></span><span class="token punctuation">}</span> <span class="token operator">&lt;</span>input <span class="token keyword">type</span><span class="token operator">=</span><span class="token string">"text"</span> onChange<span class="token operator">=</span><span class="token punctuation">{</span>handleChange<span class="token punctuation">}</span> <span class="token operator">/</span><span class="token operator">></span> <span class="token operator">&lt;</span><span class="token operator">/</span>label<span class="token operator">></span> <span class="token operator">&lt;</span>input <span class="token keyword">type</span><span class="token operator">=</span><span class="token string">"submit"</span> value<span class="token operator">=</span><span class="token string">"Update Name"</span> <span class="token operator">/</span><span class="token operator">></span> <span class="token operator">&lt;</span><span class="token operator">/</span>form<span class="token operator">></span> <span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre></div> <p>The last step for this setup is wrapping the application in a QueryClientProvider.</p> <p><strong><em>App.tsx</em></strong></p> <div class="gatsby-highlight" data-language="ts"><pre class="language-ts"><code class="language-ts"><span class="token keyword">import</span> React <span class="token keyword">from</span> <span class="token string">"react"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> QueryClient<span class="token punctuation">,</span> QueryClientProvider <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"react-query"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token string">"./App.css"</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> NameComponent <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"./components/NameComponent"</span><span class="token punctuation">;</span> <span class="token keyword">const</span> queryClient <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">QueryClient</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">function</span> <span class="token function">App</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token punctuation">(</span> <span class="token operator">&lt;</span>div className<span class="token operator">=</span><span class="token string">"App"</span><span class="token operator">></span> <span class="token operator">&lt;</span>header className<span class="token operator">=</span><span class="token string">"App-header"</span><span class="token operator">></span> <span class="token operator">&lt;</span>QueryClientProvider client<span class="token operator">=</span><span class="token punctuation">{</span>queryClient<span class="token punctuation">}</span><span class="token operator">></span> <span class="token operator">&lt;</span>NameComponent <span class="token operator">/</span><span class="token operator">></span> <span class="token operator">&lt;</span><span class="token operator">/</span>QueryClientProvider<span class="token operator">></span> <span class="token operator">&lt;</span><span class="token operator">/</span>header<span class="token operator">></span> <span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token keyword">export</span> <span class="token keyword">default</span> App<span class="token punctuation">;</span></code></pre></div> <p>React Query is quicker to write, easier to navigate, and, now that you know how to use it, an alternative to storing and updating server state with React Redux with Redux Saga or Redux Thunk.</p><![CDATA[How to Create a Localized Date Pipe in Angular 📅]]>/2021/10/angular-localized-date-pipe//2021/10/angular-localized-date-pipe/Mon, 11 Oct 2021 00:00:00 GMT<p><strong>TLDR:</strong></p> <p>To translate dates in Angular, we can think about three approaches: setting the app global locale, passing a specific locale to Angular <a href="https://angular.io/api/common/DatePipe"><code>date</code></a> through arguments, or creating a custom <code>localizedDate</code> pipe. The <code>localizedDate</code> pipe combines Angular <code>date</code> pipe and the lib <a href="https://github.com/ngx-translate/core">ngx-translate</a>. This solution prevents us from explicitly passing the locale as an argument in the template while still being able to dynamically change the dates' locale without reloading the app. Those aspects make the <code>localizedDate</code> pipe impure like [ngx-translate]'s <code>translate</code> pipe.</p> <h2>Use a pipe in the template</h2> <p>We use pipes either in a service or a template (the HTML code). In the following example, we are using the template syntax of the <a href="https://angular.io/api/common/SlicePipe"><code>slice</code></a> pipe and the <code>uppercase</code> pipe.</p> <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>div</span><span class="token punctuation">></span></span>{{ 'Grogu' | slice:1:4 | uppercase }}<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>div</span><span class="token punctuation">></span></span> <span class="token comment">&lt;!-- output: &lt;div>ROG&lt;/div> --></span></code></pre></div> <p>This is equivalent to <code>'Grogu'.slice(1, 4).toUpperCase()</code> in Javascript.</p> <p>Sadly, there is only a short list of built-in pipes in Angular. Wait a second, could we create our own?</p> <p>That's fairly easy to do because a pipe is simply a data transformer.</p> <h2>Different ways to translate dates in a template</h2> <p>Angular provides a built-in <code>date</code> pipe. It transforms a date into a string value (eg. <code>2021-03-21</code> -> <code>Mar 21, 2021</code>). The <code>date</code> pipe uses the app locale as its default locale (eg. <code>2021-03-21</code> -> <code>21 mars 2021</code> for a french locale).</p> <p>To translate dates in a template, we can think about three approaches:</p> <ul> <li>by setting the default app locale</li> <li>by passing the locale as arguments of the <code>date</code> pipe</li> <li>by combining <a href="https://github.com/ngx-translate/core">ngx-translate</a> and the <code>date</code> pipe in a new custom pipe</li> </ul> <h3>Approach 1: Setting the default locale</h3> <p>The <code>date</code> pipe uses the <a href="https://angular.io/api/core/LOCALE_ID"><code>LOCALE_ID</code></a> to determine the default locale to use. By default, <code>LOCALE_ID</code> is set to <code>en-US</code>. We can change the app locale by <code>LOCALE_ID</code> setting in the <code>AppModule</code>.</p> <div class="gatsby-highlight" data-language="js"><pre class="language-js"><code class="language-js">@<span class="token function">NgModule</span><span class="token punctuation">(</span><span class="token punctuation">{</span> providers<span class="token operator">:</span> <span class="token punctuation">[</span> <span class="token punctuation">{</span> provide<span class="token operator">:</span> <span class="token constant">LOCALE_ID</span><span class="token punctuation">,</span> useValue<span class="token operator">:</span> <span class="token string">'fr-FR'</span> <span class="token punctuation">}</span> <span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre></div> <p>With the static string value <code>fr-FR</code>, the <code>LOCALE_ID</code> value can not be dynamically changed. Instead of statically setting the <code>LOCALE_ID</code> value, we can provide a service as a value.</p> <div class="gatsby-highlight" data-language="js"><pre class="language-js"><code class="language-js">@<span class="token function">NgModule</span><span class="token punctuation">(</span><span class="token punctuation">{</span> providers<span class="token operator">:</span> <span class="token punctuation">[</span> <span class="token punctuation">{</span> provide<span class="token operator">:</span> <span class="token constant">LOCALE_ID</span><span class="token punctuation">,</span> useFactory<span class="token operator">:</span> appLocaleService <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre></div> <p>For changes made to <code>appLocaleService</code> to be reflected in our app, we need to reload the Angular app. <code>LOCALE_ID</code> is required in Angular build process. Reloading the app may reset the app state and re-trigger HTTP requests.</p> <h3>Approach 2: Passing the locale as arguments of the <code>date</code> pipe</h3> <p>The locale used in the the built-in <code>date</code> pipe can be set through the pipe's arguments.</p> <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>div</span><span class="token punctuation">></span></span>{{ '2021-03-21' | date:undefined:undefined:'en' }}<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>div</span><span class="token punctuation">></span></span> <span class="token comment">&lt;!-- output: &lt;div>Mar 21, 2021&lt;/div> --></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>div</span><span class="token punctuation">></span></span>{{ '2021-03-21' | date:undefined:undefined:'fr' }}<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>div</span><span class="token punctuation">></span></span> <span class="token comment">&lt;!-- output: &lt;div>21 mars 2021&lt;/div> --></span></code></pre></div> <p>Same as the previous approach, the static locale can be replaced by value resolved in a service. That way, we can dynamically change the locale without reloading the entire app.</p> <p>The repetitive syntax makes this approach frustrating in the syntax.</p> <h3>Approach 3: Using a custom pipe <code>localizedDate</code></h3> <p>The internationalization library <a href="https://github.com/ngx-translate/core">ngx-translate</a> provides a <code>translate</code> pipe that can be used as follows.</p> <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>div</span><span class="token punctuation">></span></span>{{ 'my.traduction.key.for.hello-world' | translate }}<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>div</span><span class="token punctuation">></span></span> <span class="token comment">&lt;!-- output (if the current locale is EN): &lt;div>Hello world&lt;/div> --></span> <span class="token comment">&lt;!-- output (if the current locale is FR): &lt;div>Bonjour le monde&lt;/div> --></span></code></pre></div> <p>The goal here is to create a <code>localizedDate</code> similar to <code>translate</code>. It will subscribe to any change in the application locale and behave accordingly.</p> <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>div</span><span class="token punctuation">></span></span>{{ '2021-03-21' | localizedDate }}<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>div</span><span class="token punctuation">></span></span> <span class="token comment">&lt;!-- output (if the current locale is EN): &lt;div>Mar 21, 2021&lt;/div> --></span> <span class="token comment">&lt;!-- output (if the current locale is FR): &lt;div>21 mars 2021&lt;/div> --></span></code></pre></div> <p>With the <code>localizedDate</code>, we gain in maintainability in the syntaxe and by using a single internationalization library for dates and contents. As others, it has its cost, this approach adds extra computation in the app. More details next in the article.</p> <h2>Our own custom localized date pipe</h2> <p>The idea is the use the built-in <code>date</code> pipe along with <code>ngx-translate</code>. Sounds complicated? Let's figure that out!</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx">@<span class="token function">Pipe</span><span class="token punctuation">(</span><span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token string">'localizedDate'</span><span class="token punctuation">,</span> pure<span class="token operator">:</span> <span class="token boolean">false</span> <span class="token punctuation">}</span><span class="token punctuation">)</span> <span class="token keyword">export</span> <span class="token keyword">class</span> <span class="token class-name">LocalizedDatePipe</span> <span class="token keyword">implements</span> <span class="token class-name">PipeTransform</span> <span class="token punctuation">{</span> <span class="token function">constructor</span><span class="token punctuation">(</span><span class="token parameter"><span class="token keyword">private</span> translateService<span class="token operator">:</span> TranslateService</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token punctuation">}</span> <span class="token function">transform</span><span class="token punctuation">(</span>value<span class="token operator">:</span> Date<span class="token punctuation">,</span> format <span class="token operator">=</span> <span class="token string">'mediumDate'</span><span class="token punctuation">)</span><span class="token operator">:</span> string <span class="token punctuation">{</span> <span class="token keyword">const</span> datePipe <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">DatePipe</span><span class="token punctuation">(</span><span class="token keyword">this</span><span class="token punctuation">.</span>translateService<span class="token punctuation">.</span>currentLang<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> datePipe<span class="token punctuation">.</span><span class="token function">transform</span><span class="token punctuation">(</span>value<span class="token punctuation">,</span> format<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span></code></pre></div> <p>Yeah, as simple as that. In this implementation, there is:</p> <ul> <li>the declaration of the new pipe with <code>@Pipe</code> (note <code>pure: false</code>, we will come back to that later)</li> <li>the injection of the app's <code>TranslateService</code></li> <li>the <code>transform</code> method of the <code>PipeTransform</code> interface</li> </ul> <p>In the <code>transform</code> method, we are using the service syntax of <code>DatePipe</code>. Hence, we can specify the current locale during instantiation.</p> <h2>Pure vs Impure pipe</h2> <p><a href="https://medium.com/@ghoul.ahmed5">Ahmed Ghoul</a> wrote a super article about <a href="https://medium.com/@ghoul.ahmed5/pure-vs-impure-pipe-in-angular-2152cf073e4d">Pure vs Impure Pipe in Angular</a>. Pure pipes (like the built-in <code>date</code> pipe) are called only if the pipe's inputs changed. On the other hand, impure pipes are called on every change detection cycle.</p> <p><strong>The <code>localizedDate</code> can not be pure.</strong> The main purpose of this pipe is to avoid passing the current locale as input. Therefore, it has to be an impure pipe to listen to the current locale changes triggered by other components.</p> <p>For the same reason, the <code>translate</code> pipe from <code>ngx-translate</code> is impure as well.</p> <h2>A Stackblitz is worth a thousand words</h2> <p>The source code of the <code>localizedDate</code> in a concrete example can be found here :</p> <iframe width="100%" height="700" src="https://stackblitz.com/edit/angular-ivy-qjfar6?embed=1&file=src/app/pipes/localized-date.pipe.ts" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> <h2>The last word</h2> <p>The approach you may want to choose to translate dates depends on your application and needs.</p> <p>If you have a specific URL per locale and redirect the user when he changes locale, the first approach should work just fine. If you are already using <code>ngx-translate</code>, creating a <code>localizedDate</code> starts to make sense.</p> <p>To help you with your decision, here is a quick comparison between the three approaches:</p> <table> <thead> <tr> <th align="center"></th> <th align="center"><strong>Syntaxe</strong></th> <th align="center"><strong>Reusability</strong></th> <th align="center"><strong>Reload required</strong></th> <th align="center"><strong>Performance</strong></th> </tr> </thead> <tbody> <tr> <td align="center">Setting LOCALE_ID</td> <td align="center">⭐️⭐️⭐️</td> <td align="center">⭐️⭐️⭐️</td> <td align="center">YES</td> <td align="center">⭐️⭐️⭐️</td> </tr> <tr> <td align="center">Using date pipe argument</td> <td align="center">-</td> <td align="center">⭐️</td> <td align="center">NO</td> <td align="center">⭐️⭐️⭐️</td> </tr> <tr> <td align="center">Custom localizedDate</td> <td align="center">⭐️⭐️⭐️</td> <td align="center">⭐️⭐️⭐️</td> <td align="center">NO</td> <td align="center">⭐️</td> </tr> </tbody> </table> <p>Happy coding 👨🏽‍💻</p><![CDATA[How I Made My Life Easier by Letting QA Write E2E Tests? ]]>/2021/09/end-to-end-test-automation-for-qa//2021/09/end-to-end-test-automation-for-qa/Mon, 20 Sep 2021 00:00:00 GMT<p>Lately, I have worked as a web developer with a company that deals with logistics. Their business is to supervise the transport of commodities from point A to point B. The application I worked on was intended for the suppliers of this company to track their merchandise. Each persona would connect to the application and perform tasks such as declaring the dispatch or the reception of the products.</p> <p>The people who worked on the previous projects followed a V-model lifecycle. These projects were delivered with many bugs after a few years. Developers spent a couple more years trying to fix the application. Therefore, the <strong>quality of the application is one of their main focuses</strong>. We were using an agile methodology and after each sprint, all the application was tested to ensure that no bug or regression appeared. <strong>QA (Quality assurance) was responsible for the quality of the application.</strong></p> <h2>Testing the application - how to do it effectively?</h2> <p>The QA was manually testing the application. Running the full set of tests would take half a day, so the tests were performed every other week. It was not frequent enough to catch all the bugs and meanwhile, QA could not focus on their other tasks. It was not satisfying enough for the client and the QA,that is why the developer team suggested <strong>implementing end-to-end (e2e) testing.</strong></p> <p>But this idea was not perfect: in typical e2e testing, developers are writing the e2e tests. While they are implementing those tests, they are not developing features, which is detrimental for the client. On top of that, <strong>the QA,</strong> which is responsible for the quality of the application, <strong>does not understand what the e2e tests are testing.</strong></p> <p>Our challenge was finding a solution where QA would not have to test the application manually but without making testing the developers’ responsibility.</p> <h2>What if QA could write tests?</h2> <p>If the <strong>QA can write the e2e tests</strong>, the tests and the quality of the application are directly managed by the QA. The QA ensures thus the high quality of the application. It is pretty fast for the QA, because they write the test once, and <strong>it is then run automatically</strong>. And no need to wait two weeks now to run the tests anymore, they can be run every day.</p> <p>Which means developers can fix bugs faster, for they might have created the bug recently and the code is still in their mind.</p> <p>This was nice and good, but we had one major issue, for our QA did not have a developer background, yet popular e2e test frameworks, such as Cypress, <strong>are asking for knowledge in Javascript.</strong></p> <h2>Writing tests without coding (or very little)</h2> <p>A common test framework, called Cucumber, <strong>can be used without development skills</strong> and both developers and the QA had already used it. Through its language (Gherkin), it allows team members to write tests in a <strong>human-readable language</strong>.</p> <p>Cucumber can be then connected to many common testing tools, and it can be connected to Cypress thanks to <a href="https://www.npmjs.com/package/cypress-cucumber-preprocessor">cypress-cucumber-preprocessor</a>. To develop tests, all you need is a grammar of test sentences, for example, “When I log in”. <strong>You still have to code, but your sentences are reusable</strong>, so when the grammar is done right, developers are not needed for every test case.</p> <p>Our process was as follows. First, <strong>the QA would decide which test cases were to be tested automatically</strong> (critical test cases that are very tedious to run manually). The <strong>developers would then translate</strong> from Gherkin to Cypress by creating the Gherkin grammar that the QA can use.</p> <p>You'll find an example of a feature written in Gherkin by the QA and the steps definitions in Cypress (as the developer will code them). You can retrieve this example <a href="https://github.com/zabeth1/cypress-with-cucumber">here</a>.</p> <div class="gatsby-highlight" data-language="gherkin"><pre class="language-gherkin"><code class="language-gherkin"><span class="token comment"># Scenario in Cucumber/Gherkin</span> <span class="token feature"><span class="token keyword">Feature:</span><span class="token important"> Cypress click</span> </span> <span class="token scenario"><span class="token keyword">Scenario:</span><span class="token important"> Click and navigate</span></span> <span class="token atrule">Given</span> I'm on the <span class="token string">"/"</span> page <span class="token atrule">When</span> <span class="token atrule">I</span> click on <span class="token string">"Querying"</span> <span class="token atrule">Then</span> <span class="token atrule">I</span> should have <span class="token string">"querying"</span> in the URL</code></pre></div> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"><span class="token comment">// Translation of the Gherkin steps to Cypres</span> <span class="token keyword">import</span> <span class="token punctuation">{</span> Given<span class="token punctuation">,</span> When<span class="token punctuation">,</span> Then <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"cypress-cucumber-preprocessor/steps"</span><span class="token punctuation">;</span> <span class="token keyword">const</span> baseUrl <span class="token operator">=</span> <span class="token string">"https://example.cypress.io"</span><span class="token punctuation">;</span> <span class="token function">Given</span><span class="token punctuation">(</span><span class="token string">"I'm on the {string} page"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token parameter">url</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> cy<span class="token punctuation">.</span><span class="token function">visit</span><span class="token punctuation">(</span>baseUrl <span class="token operator">+</span> url<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">When</span><span class="token punctuation">(</span><span class="token string">"I click on {string}"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token parameter">label</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> cy<span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span><span class="token string">".home-list"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">contains</span><span class="token punctuation">(</span>label<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">click</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">Then</span><span class="token punctuation">(</span><span class="token string">"I should have {string} in the URL"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token parameter">url</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> cy<span class="token punctuation">.</span><span class="token function">url</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">should</span><span class="token punctuation">(</span><span class="token string">"include"</span><span class="token punctuation">,</span> url<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre></div> <h2>Make our cucumber tests run</h2> <p><strong>Our team was using Xray,</strong> a popular plugin that powers up Jira by providing tools to manage tests. As a result, the QA had to write all its test cases in XRay, which allows the QA to write manual tests or cucumber tests. In addition, Xray comes with an API that developers can use to get test cases or send tests results. Each morning, <strong>the tests were automatically pulled from XRay, run and then the results were sent to XRay.</strong></p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/9797e90106cd61960e0bad150ffa3a8b/50383/CucumberTestPage.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 97.83783783783784%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAACXBIWXMAABJ0AAASdAHeZh94AAACMElEQVQ4y5VTS27bMBTUWXuSopsuCnTfTQ/QE3TVAxToKkERI4liS7IkUpQofiRRFEl3KDuJETho8kAbjwTnvXmjYbI4N04muNnOZjLzNNtSSGNmnB/+F0nb9fmeVHVzl2ZZXleE3aZ5VbOasKKk27wq62bNSU3bu4e8Ii1g4QhWethmFW26m7tduisp46jFun5fNQWKEoa8rJqsIPuK5iVtuXjuDIa410sN8qTpOi6atgf/XijkUg3WuriVumF8GDHW4kM4gbUeSxDqhPf+8M5IrF2UGrQehmEMjxGnCsd1ahLiSXjansBoOFvE8q6ex0IQbIS2kMH78HYwWi6LS/BpjTXYO+fQH0f4xcS5xSKx4TzcMs/zNM3O+VUwK6ggg570MHa9FFJDZ45EqL6XcAGKrriocTCKELbLa+iPbfJiEn8W6zagiXfeHZd/ZLBm0SQP2z3MANl8eG3scK7UU5vk15+bD19/fPz87dOX7z9/X6W9Yg0njPf9cH/fUPhLmWH/l9GacSW0vqU7uIgLFTsbu4jRHGWYIdNa0kWuYaW6EvQLDmEHaMGlhBehS1QberYtBzIycR4Ku1feUxx1FeL5O+Nqx6GqMLPFGsb4MA+X/HTBnm+59CoYLGvS1PUipWWsKctSKdV1HSGEUjpNE+aCIhdnibSLotpslqIg19dXaZpuNpssy+gaDPXWQEVjzIuXl8CMbdtK6aCZlGIcR7kGEjw0ALTW+EfOOcflc/A/6NSFKBcvO84AAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Example of a test in Xray with the cucumber syntax" title="Example of a test in Xray with the cucumber syntax" src="/static/9797e90106cd61960e0bad150ffa3a8b/50383/CucumberTestPage.png" srcset="/static/9797e90106cd61960e0bad150ffa3a8b/1d79a/CucumberTestPage.png 185w, /static/9797e90106cd61960e0bad150ffa3a8b/1efb2/CucumberTestPage.png 370w, /static/9797e90106cd61960e0bad150ffa3a8b/50383/CucumberTestPage.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span> <em>Example of a test case in <a href="https://docs.getxray.app/display/XRAYCLOUD/Gherkin">XRay</a></em></p> <h2>How was this solution for us?</h2> <p>With this solution, <strong>developers do as little code as possible</strong>. They only need to translate the Gherkin sentences into Cypress tests, but each sentence is reusable, so the QA can write new tests without the need of development.</p> <p><strong>The QA has a complete picture of the quality of the application.</strong> They decide which test cases should be executed each day and they know immediately if a test fails.</p> <p>This process is not perfect and it still requires <strong>some maintenance</strong> from the developers, but no more than for regular automated tests. It even requires less time from developers as they are not responsible for keeping the test cases up-to-date.</p> <h2>And if you want to have your QA writing the e2e tests?</h2> <p>First, train your QA for using Cucumber. It is easy to understand, even with no development skills. Feel free to check the comprehensive <a href="https://cucumber.io/docs/gherkin/reference/">Cucumber documentation</a>.</p> <p>Next, you need to decide where the QA should write their tests. Requiring QA to use git may be excessive. Xray is a good tool as it is designed to be used with Cucumber, but it can be any tool that your QA uses to write its test sets. Ideally, the tool should support Gherkin syntax and have an API to let developers run tests automatically.</p><![CDATA[Architecture guidelines for large Angular applications]]>/2021/09/angular-container-pattern-for-large-applications//2021/09/angular-container-pattern-for-large-applications/Wed, 15 Sep 2021 00:00:00 GMT<h4>TLDR;</h4> <p>When coding an Angular app for the first time, this is dangerous to keep old habits and reproduce patterns that worked for other apps like React or Vue. Working in a very large (~100 devs) project, I know that old habits can make code of poor quality.</p> <p>I propose some guidelines inspired from Angular's recommendations and Clean Code practices that will help your team to write maintanable code and ease newcomer's onboarding!</p> <ul> <li>Put the code right where it belongs : use the power of services and pipes to unclog your components.</li> <li>Code in a reactive way your components with services</li> <li>Avoid passing inputs/outputs all around thanks to the Dependency Injection System.</li> <li>Split Services when they become too big. Do the same with modules</li> </ul> <h2><strong>Old habits that get you wrong</strong></h2> <h4>Those patterns you need to let go of</h4> <p>Angular's intimate relationship with Typescript makes it easy to put data logic everywhere. Therefore it's easy to agree that <em>dumb</em> contains view-printing, and <em>smart</em> contains fetching. It is a sound decision to improve your code readability.</p> <p>In terms of performance and seperation of duties, Angular proposes an idiomatic way to code. But many of us have been tempted to try famous patterns like <em>Container Components</em> (or <em>smart</em>), or trying to reproduce stateless Components of React and other frameworks.</p> <p>That being said, Data often plays a central role. Once again, the temptation to try famous patterns like functional components and a Redux data-store is very strong. Indeed, some don't see how to find a robust way to manage their data.</p> <p>Nevertheless, trying to mimick patterns you learned on other codebases blindly into an Angular application is such a dangerous idea and I want to show you how to make your Angular application maintanable.</p> <h4>Improving readability in consulting companies</h4> <p>I have been working here at <a href="https://www.theodo.fr/"><strong>Theodo</strong></a> (also in the <a href="https://www.theodo.co.uk/">UK</a>) with teams that encoutered a huge developers turnover. Developers rarely stayed more than one year on the same code base. I even found myself switching teams every 2 months.</p> <p>Therefore, it is crucial to choose a pattern that will help new developers to quickly switch codebases. That's why I want to introduce you to <strong>the right way</strong> of thinking your Angular Application Architecture. This approach makes small use of other famous <em>patterns</em> and mainly explains Angular's recommendations.</p> <h2><strong>Guidelines for clean and efficient Angular coding</strong></h2> <h4><strong>Use angular-cli</strong></h4> <p>These are the rules you can establish in your team to make you Angular application cleaner. And as a quick rule, I advise you to always use <code>angular-cli</code> when creating new components and services in your app, so you're not tempted to write less.</p> <h4><strong>You are writing presentation code in .ts or .html ? Make a pipe</strong></h4> <p>If the data needs simple mappers, create <code>pipes</code> that you will use in component's template. Indeed those mappers add <em>usually <strong>nothing</strong> but noise</em> to your code, hide them away !</p> <p>Don't write functions for data transformation in your component's .ts, and don't write complex functions in your component's html template. Code smells :</p> <ul> <li>You call functions in your template (<a href="https://medium.com/emoteev-blog/call-functions-in-angular-templates-the-good-way-f025f65b0e55">See why that's a bad idea</a>)</li> <li>You write a lot of logic in {{ your template }}</li> </ul> <p>Here is an example of a <strong>too-intelligent</strong> component. It has functions in its template, with logic...</p> <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>ul</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>li</span> <span class="token attr-name">*ngFor</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>let member of team.member<span class="token punctuation">"</span></span><span class="token punctuation">></span></span> {{ isUserFrench(member) ? '🇫🇷' : '🚩' }} Star: {{ isUserCapableOneOfTheBest(member) }} <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>li</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>ul</span><span class="token punctuation">></span></span></code></pre></div> <p>You can put everything in a pipe, a test it seperately !</p> <div class="gatsby-highlight" data-language="ts"><pre class="language-ts"><code class="language-ts">@<span class="token function">Pipe</span><span class="token punctuation">(</span><span class="token punctuation">{</span> name<span class="token operator">:</span> <span class="token string">'memberBasicInfoList'</span> <span class="token punctuation">}</span><span class="token punctuation">)</span> <span class="token keyword">export</span> <span class="token keyword">class</span> <span class="token class-name">MemberBasicInfoListPipe</span> <span class="token keyword">implements</span> <span class="token class-name">PipeTransform</span> <span class="token punctuation">{</span> <span class="token function">transform</span><span class="token punctuation">(</span>team<span class="token operator">:</span> Team<span class="token punctuation">)</span><span class="token operator">:</span> <span class="token punctuation">{</span>flag<span class="token operator">:</span> <span class="token string">'🚩'</span> <span class="token operator">|</span> <span class="token string">'🇫🇷'</span><span class="token punctuation">,</span> isStar<span class="token operator">:</span> <span class="token builtin">boolean</span><span class="token punctuation">}</span><span class="token punctuation">[</span><span class="token punctuation">]</span> <span class="token punctuation">{</span> <span class="token comment">//calculate flag and isStar</span> <span class="token keyword">return</span> <span class="token punctuation">{</span>flag<span class="token punctuation">,</span> isStar<span class="token punctuation">}</span> <span class="token punctuation">}</span></code></pre></div> <p>Then you have a very readable template !</p> <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>ul</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>li</span> <span class="token attr-name">*ngFor</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>let memberInfo of team | memberBasicInfoList<span class="token punctuation">"</span></span><span class="token punctuation">></span></span> {{ memberInfo.flag }} Star: {{ member.isStar }} <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>li</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>ul</span><span class="token punctuation">></span></span></code></pre></div> <h4><strong>Make reactive Services that support your components business logic</strong></h4> <p>Services are your intelligent blocks of code. If you wanted to put code in a <em>container</em> component, put it here ! Here are some some rules of thumb :</p> <p>Wrap all the <strong>data fetching in a service</strong>, and inject them in the component(s). This service should expose ready to use observables. It should also give methods to send data from components to the service. Same rule applies if a service becomes too big : split it and inject one service into the other service.</p> <p>Read more about :</p> <ul> <li>Tarun Sharma's <a href="https://medium.com/tkssharma/reactive-pattern-for-data-services-in-angular-859e4ca39099">Reactive Pattern for data services in angular</a></li> <li>Paul Clavier's <a href="https://www.sipios.com/blog-tech/reactive-programming-rxjs-observables-angular">Reactive Programming with rxjs observables</a></li> </ul> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/69163b704e33be5466175ffee97cea37/50383/reactive-pattern.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 60%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAIAAADtbgqsAAAACXBIWXMAAAsSAAALEgHS3X78AAABcklEQVQoz21SCW7DIBD0/19XqVLVNknjJL5tsLnBB+5gZCuNao0RsDvsMEsySNMLQB+gXHeDKLshb2jTc0wqygjXL2lYJi9kpoy2E+l5mpXnW1Y03S2vsrIR2jJlt+T/yJgAyrp1/5xzyzLHuV9XylU7iEiLSPBjdxAa8u5V13OF1EHq77T4uubA+V5ZN2KzJiyrCe4ySNvzvTL0cO0QaHohtUWe0jbNm2tWpVl9L9tpCvWFcSibNSSIlUFpQpjKG4JTi7ZHTGin7WiR7ldgwbh440ZlRhREMtIGZSnfZBMuL4/i/fNyuuU/D4QE4QredkxFgIAlWgAyyrx9nNK8ik4lTAbNXCNAQcNEmhH1ebiLDeMOEAiTD/hyuL0Zppk0uAzqUia0MdMMyav3Pvq8jR78KJsF2bth8BbtxS4EZVUL8jzPy7IQ2jPOj87VlKFsJMe+Hn0OR2x9Hn3wyYP9jGlecMGnJv95JMfTiQf9g5cc4Bekkqojd9zDMQAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Illustration of the reactive pattern" title="Illustration of the reactive pattern" src="/static/69163b704e33be5466175ffee97cea37/50383/reactive-pattern.png" srcset="/static/69163b704e33be5466175ffee97cea37/1d79a/reactive-pattern.png 185w, /static/69163b704e33be5466175ffee97cea37/1efb2/reactive-pattern.png 370w, /static/69163b704e33be5466175ffee97cea37/50383/reactive-pattern.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h4><strong>Code in a reactive way with observables as inputs and funtions as outputs</strong></h4> <p>Use observables in your components as if they were already containing the data you want, because that is not the role of component to update the data. Then build <strong>public</strong> methods in your service to <strong>notify</strong> it of the last events. Your component should contain very minimum logic.</p> <blockquote> <p>Think of your component as a puppet of the service. It shows the data the service wants. It tells the service when something happens.</p> </blockquote> <h4><strong>Split your services when they become bigger than 100 lines</strong></h4> <p>Put business logic as soon as possible in services and prepare the data inside of them to be displayed. For example a <code>TeamMember.service</code> that relies of <code>TeamsStore.service</code> and for example another <code>User.service</code> (that can come from somewhere else than TeamsModule)</p> <p>To split your code, detect parts of code that are heavily linked to each others.</p> <h4><strong>When your base module becomes too big and handles many topics : split into smaller modules</strong></h4> <p>Put common business items in a module, for example "TeamsModule". Split them conviniently with Angular routing as soon as possible. When ?</p> <ul> <li>Some parts become more and more independant, and some parts are never displayed together</li> <li>Your module is too big for a new-comer to understand it</li> </ul> <p>Your parent/children modules can communicate through common services. Learn more about when to split modules here : <a href="https://michelestieven.medium.com/organizing-angular-applications-f0510761d65a">Angular: Understanding Modules and Services</a></p> <h4>Let's study an example: Breaking down one component into smaller simpler pieces</h4> <p>We talked about writing reactive services, and breaking down big chuncks of code into smaller meaningful one. This example shows a typical example when you have a very big component that needs to be refactored. I often found myself in this exact situation.</p> <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">InviteTeamMemberComponent(httpClient: HttpClient, errorService: ErrorService, router: Router) A. Can call the team-backend and POST a new team member B. Will show an error message (in global snackbar) if failed C. Will redirect to Team dashboard if successful D. Creates FormGroup for team member informations (also stores the data and validates it) E. Displays the data in the template F. Keeps track of the loading state of the query</code></pre></div> <blockquote> <p>That's a lot of responsibilities, huh ?</p> </blockquote> <p>Let's create a client that can do http-calls, especially the POST (task A.)</p> <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">TeamMemberClient(httpClient: HttpClient) A. Can call the team-backend and POST a new team member</code></pre></div> <p>and inject it in another service which has the responsibility to manipulate data (tasks B, C and D).</p> <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">InviteTeamMemberService(teamMemberClient: TeamMemberClient, errorService: ErrorService, router : Router) B. Will show an error message (in global snackbar) if failed C. Will redirect to Team dashboard if successful D. Creates FormGroup for team member informations (also stores the data and validates it)</code></pre></div> <p>that we inject inside the initial lighter component that does the two last UI features (tasks E and F)</p> <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">InviteTeamMemberComponent(inviteTeamMemberService: InviteTeamMemberService) E. Displays the data in the template F. Keeps track of the loading state of the query</code></pre></div> <h4><strong>Avoid Input/Output and prefer injecting specialized services</strong></h4> <p>You can sometimes avoid <a href="https://gopi-it.medium.com/angular-input-output-and-services-for-state-management-d6fec85bcc3f">Input/Output drilling (more specialized article)</a> with either service or <strong>content</strong> injection, this will look like a simple advice to Redux advocates, but here is the good news : we don't need redux !</p> <p>Here is an example of input drilling where a 2 level deep button opens a modale upper in the hierarchy, all managed by a <em>smart</em> ancestor component !</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 735px; " > <a class="gatsby-resp-image-link" href="/static/5f429a533ea163e7a420bd31adbbdf5e/7608e/input-drilling.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 90.27027027027027%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAASCAYAAABb0P4QAAAACXBIWXMAAA7EAAAOxAGVKw4bAAABrklEQVQ4y6WTa1PqMBCG8/9/3Pmqx1ERBJFCW3JP05bX3SiQllY9czqz00ySffLuTRyNx5TV2g3WuY3PcxPfwc4AGxrEtkOILZQN1/MJsJhTeAb6JiL/TqfTBTqlWPwUMgNi28L5AB9Cgjaxo3MPSeBx+OK7vEnrE8A6B20MlNJoCV5LjU1R4XDUKKUZpEfMKePXS2XRdl1S+efuGbtSIsaI1VuBp9UWL5sddpUchC7milHUCgtyKEmFcQH3izUKcra+SftLhr5uIem+a9ok4EZhDlxt98nhmYz/NkRo16SH3g91On9cbiC1gaf8ajtSmMP4Nc7flhw5tLd9hYrCz9PBf0V3DmWFp8USxthUfTEuBP/brv+sJhWAk65I2eedYYMzoOtP6Ps+9ekl5Fwd50NpjWJfwjkPQzmr1O3UXP0I7K59KfJDVsNhhq9mbmgyxg/OTdOlD9MGLbhFHl7WeFi84i8ZV1TmYzYxu+O9gULOW6Bw17tD6q99rScV/GQJyEPP0+C9T+vjV5X/BXQDDDSn3E/nas3l6FdAXkQKmcOWM0X4NfDsyK3B9j8wtg8zZ2gHJBd53wAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Example of input drilling" title="Example of input drilling" src="/static/5f429a533ea163e7a420bd31adbbdf5e/7608e/input-drilling.png" srcset="/static/5f429a533ea163e7a420bd31adbbdf5e/1d79a/input-drilling.png 185w, /static/5f429a533ea163e7a420bd31adbbdf5e/1efb2/input-drilling.png 370w, /static/5f429a533ea163e7a420bd31adbbdf5e/7608e/input-drilling.png 735w" sizes="(max-width: 735px) 100vw, 735px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>Valuable code is <strong>spread</strong> among all the components and useless Input/Output mechanism pollutes the intermediary. On top of that, the ancestor component contains the data, and has the responsibility to update it. Here's what you would prefer :</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/978d469c215f025c8a676b68092e2b1f/50383/service-pattern.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 72.97297297297297%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAPCAYAAADkmO9VAAAACXBIWXMAAA7EAAAOxAGVKw4bAAACK0lEQVQ4y31TyW4TQRScX0BCQnwTPxDxOXwBd5DgBhJ7JJQLB26ccuGSEGMrzmQW27P1Nj2Lx8V7bbcZOwRLz7P0dFW9qtdBqWpU2oKv43t/VbZFvx5gu/6f3xxXwH+FNBCmcXW8wTYNGlctdM3r9r+gQbQsMJlHiBY5klV5oFYYC6U1ykpAKoVS6h25vVdtsCwkloVAmlfge/8Bq12VCpI2bzZASy0zWEykWaX/djMC4+eA2Xy7XIVjtm5TvFO8IMIwzfcgTM7vvFIm8ioDL92XoRBM0zmwYsTuwbmEI5GIVgVU3aLtB2jbbRWOfWCwYb3GMAxoONWRn7MwwTxe7JVICsgpqySEoJLKER0A1pRklhfIi5LAt+1zm7zO4U1vU/eOgdj3mBQy2GweQhKgJkHBtg3fWk3qKAH6sdnzeIVFLpAJTYpaFJlAb3vcLjJcTOdONRNvKLW2Xx966ENwz6SAF1nF9CZ2VQqDD7+v8PbqAmlWuQm4SbaEcheW85B9Y03K2P04+GKfBJ8UUvX85zkevn+JB+9e4Nn5D2TLEgmlfWcOe5LKw7vZDLRgENKAX0epa2cWxrhOltCVwcn3Mzz69AqPP7/Gk2+nsKZFaQ7BnMKu71GRsV3XOVNzapdNz3fF/ilq9yO1+/TsFCdfv+DN5BItqc531oxVBoraWlMQddPde+DdmaY5m0wj/JqEqCkgf0qOj2BwPOnHg+7LDTmPUN3sB77Ud8/zH00DcOYHgh8cAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Solving input drilling with a service" title="Solving input drilling with a service" src="/static/978d469c215f025c8a676b68092e2b1f/50383/service-pattern.png" srcset="/static/978d469c215f025c8a676b68092e2b1f/1d79a/service-pattern.png 185w, /static/978d469c215f025c8a676b68092e2b1f/1efb2/service-pattern.png 370w, /static/978d469c215f025c8a676b68092e2b1f/50383/service-pattern.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>This was an example of refactoring using service. You can also avoid one layer of Input/Output<a href="https://angular.io/guide/content-projection"> with content projection</a>. In the example above, you could get rid of <code>@Output() edit</code> if you directly inject the button of the action.</p> <h2><strong>What are the advantages of doing so ?</strong></h2> <ol> <li><strong>Synchronization within your team</strong> : You can still use old patterns, but don't make it a must ! Just communicate on your rules with your team and write coding guidelines that every one of you has read. You'll see tremendous improvement very quickly</li> <li><strong>Simpler hierarchy</strong> : your DOM is not polluted with useless container components. CSS is simpler, and inspecting the application is simpler too ! You can understand it faster ! If other teams use the same coding guidelines, you'll be able to switch teams easily.</li> <li><strong>Easier onboarding,</strong> as a new developer you know where things are by opening a package : services, components and pipes are doing what you expect them to do ! Also, less wrappers means less clicks to get to meaningful code !</li> </ol><![CDATA[AWS Fargate: harness the power of serverless for long-running computational tasks]]>/2021/09/serverless-for-long-computational-tasks//2021/09/serverless-for-long-computational-tasks/Tue, 14 Sep 2021 00:00:00 GMT<p><strong>TLDR:</strong> <em>We all know the advantages of serverless computing: pay only for what you use; scale up and down with ease; abstract away the complexities of managing servers. In this article, I illustrate how you can acquire these benefits for long computational tasks which are too large for Lambda or Google Cloud Functions to handle. I share a case study from a recent project and how we used AWS Fargate (along with S3, Lambda, DynamoDB, and more!) to create a robust task architecture.</em></p> <p>Recently, a client approached us with a state-of-the-art modelling system which had been designed by industry-leading domain experts. The techniques used in the modelling software meant that, for its use-case, the software could run simulations in an order of magnitude less time than the competing solutions.</p> <p>Despite being an industry-leading scientific model, the delivery of that solution through software was holding the product back from its full potential. The software interface was a Python GUI and was distributed as source code directly to users. The model could take hours to run and if it failed (often due to invalid input), the modeller would only discover this error on returning to the program after several hours away, only to correct the error and rerun the test case. Often, multiple variations need to be simulated, however the desktop application couldn't execute multiple runs concurrently (as the software is limited by the processing power of the machine it runs on).</p> <h1>The mission</h1> <p>We wanted to bring this software to the market as an industry-leading product in terms of scientific modelling <em>and</em> software engineering. We would leverage cloud services to mitigate some of the disadvantages of the desktop implementation.</p> <p>Here are some of the main things we wanted to add to the current functionality of the model:</p> <ol> <li> <p>Improved distribution by migrating to the cloud:</p> <ul> <li>Easier on-boarding (no software installation - just login).</li> <li>Improved security (no user access to source code).</li> <li>Run the model from any computer.</li> </ul> </li> <li> <p>Increased scalability:</p> <ul> <li>Execute multiple model runs concurrently to save users' time.</li> </ul> </li> <li> <p>Improved user experience:</p> <ul> <li>More data validation rules to prevent failed model runs.</li> <li>Email notifications for when the model has completed a run.</li> </ul> </li> </ol> <h1>The challenge</h1> <p>The abstract problem to solve was that we had a 500,000-line black-box program that was tightly coupled to the environment and file system of the machine it was running on. In our case, three issues stand out:</p> <ol> <li>The model code is tightly coupled to the modeller's development environment - it calls <code>os.system</code> in many places to run shell commands on specific libraries (these included dependencies on GNU Fortran - not something you see in your everyday web application!).</li> <li>The model code is tightly coupled to the modeller's file system - it creates intermediate output files to save data between different steps in a run. This decision was initially made from a scientific standpoint - a modeller might want to check some intermediate output - and had stuck as a design pattern across the software.</li> <li>The model code has significant complexity, and re-engineering it came with risks to the validity of the results.</li> </ol> <p>These constraints meant that we needed:</p> <ol> <li>To run the model code on a system with the same environment as the modellers' computers.</li> <li>A way to persist intermediate files in the cloud between different runs.</li> </ol> <h1>The strategy</h1> <p>The high-level flow of a modeller executing a model run would go like this:</p> <ol> <li>The modeller goes to the web application from their browser and configures a new run. Parameters that they configure are saved to DynamoDB and input files they upload are saved in S3.</li> <li>The modeller clicks the "execute" button for a specific task, which triggers (a Lambda to trigger) a task in Fargate. This spins up a Docker image registered in ECR with the correct environment and source code to run the model. The task name is passed in as an environment variable.</li> <li>The task pulls the correct input files from S3 and the user-defined parameters from DynamoDB.</li> <li>The model task is then executed as it would be on a modeller's local computer.</li> <li>The model syncs the resulting output directory to S3 and EventBridge and SES are used to notify the user of the status of the run via email.</li> </ol> <p>The architect of the project drew a handy architecture diagram to illustrate exactly how we achieved these steps and I'll go through the steps that we took along the way.</p> <br> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/626f8d4a6db3c24cff8e24b633d69c3a/50383/task-architecture.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 59.45945945945946%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAAAsSAAALEgHS3X78AAACDUlEQVQoz3VT227TQBD1L/ETfAlv/EAfeIB3ClIfeGpVUSQkRKKCBC00astFLRKoapWokUiapI2di0MSx47t9V7sw15cbAUYaaTZ2dkzZy5rIZcsy1C2y6okmIXgVPx1v/rWKjvcyQS+HxTAMH4mKHa+buD8+lsOkGKVyNzzsFj4sJSDCxPwrHqCynFd24QkkhGH4AJzf44Hlft4f1bVd6kEvGUo8rebb0/xsnYhAdMUqTCldOwJ+qOptnt9G237CstwqVVwLpPEiKUywUAZk+8kcA7YdVw4rmdKNtzTvEwjQRzgpPdFs2t1rqBS9gYjuL/GuhVREqHoXCEWOf6M+KAGBbf88RxRY1fbio2CT+Qg/IkDevoYy+Y7EKpYCSxZDFK/Bvve1smiRhW0tQ/L2z+AW32js7V212F/emHAw0APQ9ujDvpP7mJW2wBTQyIEISg6T6tor23pmJutexhX1mDxMARZLMwgOJBvBihnIIzo8jin6DbP4c9cnVikHDxjcIYD3Q4lQ3cBdxpKhts78Da3S33MV0U2nVCCiCYIZfmepEbkFUli7VdqVsbE9y8/YFB/DYtNZ1D6r4WVo5Xl3oB6I/iHDyF6R7fLV8SnZpjO5R7GjVfFlP9sfclO5STnP8+QjNvoPrqD6cf1vBBRJF/9KatfqHzmcve6skdh4MNuXmA27P/3a2Z62VP8BlJDkcFEDs53AAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="cloud infrastructure architecture diagram" title="cloud infrastructure architecture diagram" src="/static/626f8d4a6db3c24cff8e24b633d69c3a/50383/task-architecture.png" srcset="/static/626f8d4a6db3c24cff8e24b633d69c3a/1d79a/task-architecture.png 185w, /static/626f8d4a6db3c24cff8e24b633d69c3a/1efb2/task-architecture.png 370w, /static/626f8d4a6db3c24cff8e24b633d69c3a/50383/task-architecture.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <br> <h2>Dockerize the tasks: Fargate + ECR + VPC</h2> <p>The first step was to create an environment to run the model; Docker was designed to solve this very problem ("well, it worked when it ran on <em>my</em> computer!").</p> <p>A new container was defined and the required dependencies for the project were resolved (we used <a href="https://python-poetry.org/">Poetry</a> to manage the Python libraries).</p> <p>We used Docker volumes for the project so that any intermediate files created within the container are synced with our outer file systems to inspect and validate. A <code>docker-compose.yml</code> file was also written to simplify bringing the container up and down and passing in environment variables from a <code>.env</code> file.</p> <p>There are several discrete tasks that the model can run, each one generating different output files. For our use-case, we would have a single Docker image, which contains the model code for every type of task - the task to be executed would be defined as an environment variable (a container is designed to be brought up with a single purpose, then taken down once the task is complete).</p> <p>Now that we know the task we want to execute, we can define the functions we want to run for each task. We define a <code>TASK_DEFINITIONS</code> variable which stores handlers for each task: <code>preprocessor</code>, <code>executor</code> (where we will run the main task), <code>postprocessor</code>, and <code>failure</code> (the model can be temperamental to user input, and we need to handle failure explicitly). I will come back to the pre- and post-processors in the next section.</p> <p>The executor is simply a function which makes a call to the model source code to execute the desired task. This means we can define an <code>env</code> variable - <code>TASK_NAME</code> - and run the model locally by accessing bash from within the container:</p> <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">docker-compose up -d docker-compose <span class="token builtin class-name">exec</span> <span class="token operator">&lt;</span>image-name<span class="token operator">></span> <span class="token function">bash</span> python main.py</code></pre></div> <p>Note: my local machine doesn't have the correct dependencies to run the model code, but the container that I run on my local machine <em>does</em>.</p> <p>The container can now be migrated to the cloud so that the model code can be accessed from anywhere in the world without the need to manually distribute the source code.</p> <p>The image is registered with AWS Elastic Container Registry (<a href="https://aws.amazon.com/ecr/">ECR</a>) where the container can be stored and accessed (<em>Model registry</em> in our architecture diagram). A Virtual Private Cloud (<a href="https://aws.amazon.com/vpc/">VPC</a>) is configured to accept requests to trigger an AWS <a href="https://aws.amazon.com/fargate/">Fargate</a> task.</p> <p>Fargate is "<em>serverless compute for containers</em>"; this means that our container isn't running in the cloud when we don't need it - AWS handles the provisioning of resources to "spin up" a container from our image at the moment we want to run a task. This means that we are only paying for what we use whilst still having the potential to scale ad infinitum (we can trigger multiple tasks concurrently and AWS will just spin up more instances for us in the cloud). The one drawback of this architecture is that Fargate requires some time to build the container when we request for our task to be run (this was around a minute for our (large) container).</p> <p>You can see Fargate (<em>Model task</em>) sitting at the centre of our architecture diagram within the VPC in ECS.</p> <p>We also set up <a href="https://aws.amazon.com/cloudwatch/">Cloudwatch</a> to create logs for each run to help us and the modellers debug issues with the model.</p> <h2>Syncing input and output directories: S3</h2> <p>Until now, the model works in the cloud so long as the required input files are present in the file system of the Docker image; however, input files are specific to the context of a run, and if we want to get meaningful output then we need to pass in the correct input files at the point that we execute a task.</p> <p>For each task, a new instance of the image is spun-up, so any intermediate files are lost when the container is pulled down and the context is lost (this is fine, as we want different executions to remain distinct from each other).</p> <p>We set up an <a href="https://aws.amazon.com/s3/">S3</a> bucket to store the inputs and outputs of tasks, which means that the files which we expect to persist between different tasks can be synced between permanent storage from task to task.</p> <p>Each task was saving its output to a different directory, so within our <code>TASK_DEFINITIONS</code>, we define an <code>input_dir</code>, from which we would <em>pull</em> the contents from S3 and a number of <code>output_dirs</code> to which we would <em>push</em> the output of the model. These outputs might be downloaded by the user or used by a different task as input files. This syncing of files happens in the <code>preprocess</code> and <code>postprocess</code> of each task.</p> <h2>Data model: DynamoDB</h2> <p>As well as input files, the model also depends on a range of input parameters that the user can define. In the desktop app, the program gets these input parameters by reading and parsing a local file called <code>INPUT_PARAMETERS.txt</code>.</p> <p>Each model run entity has a corresponding entity within a DynamoDB table (in practice there are multiple types of entity available for the user to manipulate). The database follows the single-table design as described in <a href="https://www.dynamodbbook.com/">The DynamoDB Handbook</a> by Alex DeBrie and uses <a href="https://github.com/jeremydaly/dynamodb-toolbox"><code>dynamodb-toolbox</code></a> to model entities. The user can interact with these entities via the web application to set properties of the run.</p> <p>We created a corresponding manager in the dockerized model code for each type of entity in the database (e.g. <code>RunManager</code>) which can get the entity and also perform simple updates on text fields (set error messages etc.). The entity is retrieved from DynamoDB which then overrides the variables which were previously being retrieved from <code>INPUT_PARAMETERS.txt</code>, which otherwise describes a generic run.</p> <p>The model is now no-longer dependent on a local file for input of discrete parameters; but how does the user configure those variables to get a final valid output to their desired parameters?</p> <h2>Web Application: Lambda + Next.js</h2> <p>It's now time to plug the main pieces together.</p> <p>We developed a Next.js frontend to interface with the remote model. This comes with all the advantages of developing UI using technologies that were designed to handle UI (i.e. not Python!). This meant our input methods were more interactive and our outputs more visual, as we were able to leverage existing libraries.</p> <p>We added validation rules to the inputs (with warning messages) to prevent the user running the model with parameters that don't make sense. We accept input files as upload fields of several forms, which are pre-processed to validate that the values in the files were within the expected range before uploading to S3.</p> <p>We built our backend REST API using AWS <a href="https://aws.amazon.com/lambda/">Lambda</a>. As with Fargate, Lambda offers flexible scaling, has a <a href="https://retool.com/integrations/aws-lambda">nice integration</a> for the back-office tool we created in <a href="https://retool.com/">Retool</a>, and we didn't mind the infamous <a href="https://aws.amazon.com/blogs/compute/operating-lambda-performance-optimization-part-1/">cold starts</a> for our use case.</p> <p>Another advantage of using Lambda was that we can use AWS <a href="https://aws.amazon.com/cloudformation/">CloudFormation</a> (infrastructure as code solution) to provision our infrastructure from a <code>serverless.ts</code> file which defines all the resources required for an environment. If we wanted to create a new environment, running <code>sls deploy &#x3C;env-name></code> would provision the resources defined in the file instead of needing to interact with the AWS console.</p> <h2>Notifying model outcomes: EventBridge + SES</h2> <p>Until now, a specific model task can be triggered to run in the cloud where it will sync the relevant input files, pull the correct input parameters, run the task, and sync the output files back to S3. Finally, we need to notify the user of the outcome of the model (success + results/ error + logs).</p> <p>In the model, we defined a <code>RunEventsManager</code> which could send an event via AWS <a href="https://aws.amazon.com/eventbridge/">EventBridge</a> when the <code>postprocess</code> or <code>failure</code> handler was completed in the model. We added a Lambda in our backend which was triggered by an EventBridge event (because we were using CloudFormation, we just needed to define the event that triggers the lambda in <code>serverless.ts</code>).</p> <p>When an event occurs, the Lambda uses AWS <a href="https://aws.amazon.com/ses/">SES</a> client to send an email to the user, which notifies them immediately when they can check their results - there's no need to come back to check that the model hasn't failed every 15 minutes anymore!</p> <h1>Summary</h1> <p>Using the described architecture, we had delivered a significant improvement to the user experience in running scientific modelling tasks by bringing the software to the cloud and affording it all the advantages of a serverless implementation.</p><![CDATA[Rendering Blender scenes in the cloud with AWS Lambda]]>/2021/08/blender-serverless-lambda//2021/08/blender-serverless-lambda/Sat, 28 Aug 2021 00:00:00 GMT<p>You can fairly easily leverage AWS Lambda to render scenes with Blender. It can make sense to use Lambda functions if you need to render a large number of assets in little time and if each asset is simple enough to be loaded and rendered in reasonable time (within the maximum lifetime of a Lambda) given the limited resources offered by Lambda functions (maximum 6 vCPUs and 10GB of RAM at the time of writing). If you have to render more complex assets, going with EC2 instances or AWS Thinkbox Deadline is a better solution.</p> <p>When I came up with the idea of running Blender on Lambda, I naturally checked if someone was doing it somewhere and <a href="https://twitter.com/awsgeek/status/1108049045635776512">found out that @awsgeek had done something similar in 2019</a> so at least I knew it was possible. What we are going to present in this article slightly differs from his approach though.</p> <h2>The architecture</h2> <p>Here is the very simple architecture that allows you to render Blender scenes on AWS Lambda stored on an S3 Bucket. The final renders are stored into the same bucket.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/7bcfb03e2c0caef7ff34b11aa99490b0/50383/blender-serverless-lambda-architecture.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 60.54054054054055%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAAAsSAAALEgHS3X78AAACI0lEQVQoz3VTy27TQBT1H/ANLLOp2PIHIGXDij9gDUgsiJDKEglYhVUQbaWCKhVViC1Cal0gQVRQmoRCGnCC4jp2HDvO+P08zIxxlCbKlUdj35k599xzxgJoZFm2NJIkQRzHdE7h2RZcYiFJU0RRyI6gNfiCe3s3cXv3Bt61X/NckiYQisMFEIuULriuA9/3EKWA3hYxPNlHlGSwHQLEwGH7Pa49WcP1p1ew8/EFsgjw6H4Bc8EAwzDAhDgwLcK/bduGPtIwMQ26Fs72SqMuHuzcwd2tW2icHfKc57kQWFuVSgWdToe2GOHb8Qk6x5/QPhLRbLUR+D4URcF4bHAJisKgj+fQ4ibhOWtqwTAMCKzdUqmERqNBGUTo9yRY9RqU+iucdX/PZJjv4v8bNFWBNjznX/3+X/w8PYUgiiLK5TJqtRqGwyGq1Wd4s7eLl5vPsbGxSXX0L4Dl7DJQaaEaDs7HTs7QmkDTNAhMI1VVYZomNSOlAAEIzVlTQh2NOACbWScFMx72AMFRFWR/HdC+8hTTn7vsuu5MH25/krvMcowh01eSJG5KQIUPKab5/S1Gjy5jtH4J5OAxAnrOd+1ll4vBwBgzNmRZhq7reY6yYFytHwfoPbyKP/fXYHzYZjcJWRLnDOfv4KIJq8I2x5Cbn6E065iocsFomeGqP2exsONRKXoyfkkDGJMp1zZbBFwVFwul+SVWu2hRbcUeQU8nSLLcsH+nKILL0UcWfAAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Architecture diagram of a serverless solution to render Blender scenes with AWS Lambda" title="Architecture diagram of a serverless solution to render Blender scenes with AWS Lambda" src="/static/7bcfb03e2c0caef7ff34b11aa99490b0/50383/blender-serverless-lambda-architecture.png" srcset="/static/7bcfb03e2c0caef7ff34b11aa99490b0/1d79a/blender-serverless-lambda-architecture.png 185w, /static/7bcfb03e2c0caef7ff34b11aa99490b0/1efb2/blender-serverless-lambda-architecture.png 370w, /static/7bcfb03e2c0caef7ff34b11aa99490b0/50383/blender-serverless-lambda-architecture.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h2>The files you will need to run this "at home" (in the cloud)</h2> <p>To run this on your own AWS account, you will need four files:</p> <ul> <li><code>Dockerfile</code> describes the container image that each lambda function execution is going to run</li> <li><code>serverless.yml</code> describes the infrastructure above</li> <li><code>app.py</code> handles the event from the Lambda infrastructure and starts up blender</li> <li><code>script.py</code> manipulates Blender from the inside to lad a scene and render it.</li> </ul> <h3>File structure</h3> <p>Here is how these files should be arranged:</p> <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">project directory +-- Dockerfile +-- serverless.yml +-- app | +-- app.py | +-- script.py</code></pre></div> <h3>Dockerfile</h3> <p>For the <code>Dockerfile</code>, I started from a great set of <a href="https://github.com/nytimes/rd-blender-docker">Docker images for Blender</a> maintained by the R&#x26;D team of the New York Times. We're using the latest image that runs Blender rendering with CPU only (as Lambdas don't have GPU capabilities unfortunately).</p> <div class="gatsby-highlight" data-language="dockerfile"><pre class="language-dockerfile"><code class="language-dockerfile"><span class="token keyword">FROM</span> nytimes/blender<span class="token punctuation">:</span>2.93<span class="token punctuation">-</span>cpu<span class="token punctuation">-</span>ubuntu18.04 <span class="token keyword">ARG</span> FUNCTION_DIR=<span class="token string">"/home/app/"</span> <span class="token keyword">RUN</span> mkdir <span class="token punctuation">-</span>p $<span class="token punctuation">{</span>FUNCTION_DIR<span class="token punctuation">}</span> <span class="token keyword">COPY</span> app/* $<span class="token punctuation">{</span>FUNCTION_DIR<span class="token punctuation">}</span> <span class="token keyword">RUN</span> pip install boto3 <span class="token keyword">RUN</span> pip install awslambdaric <span class="token punctuation">-</span><span class="token punctuation">-</span>target $<span class="token punctuation">{</span>FUNCTION_DIR<span class="token punctuation">}</span> <span class="token keyword">WORKDIR</span> $<span class="token punctuation">{</span>FUNCTION_DIR<span class="token punctuation">}</span> <span class="token keyword">ENTRYPOINT</span> <span class="token punctuation">[</span> <span class="token string">"/bin/2.93/python/bin/python3.9"</span><span class="token punctuation">,</span> <span class="token string">"-m"</span><span class="token punctuation">,</span> <span class="token string">"awslambdaric"</span> <span class="token punctuation">]</span> <span class="token keyword">CMD</span> <span class="token punctuation">[</span> <span class="token string">"app.handler"</span> <span class="token punctuation">]</span></code></pre></div> <h3>Serverless configuration file</h3> <p>The <code>serverless.yml</code> file is fairly simple:</p> <div class="gatsby-highlight" data-language="yaml"><pre class="language-yaml"><code class="language-yaml"><span class="token key atrule">service</span><span class="token punctuation">:</span> blender<span class="token punctuation">-</span>on<span class="token punctuation">-</span>lambda <span class="token key atrule">frameworkVersion</span><span class="token punctuation">:</span> <span class="token string">'2'</span> <span class="token key atrule">configValidationMode</span><span class="token punctuation">:</span> error <span class="token key atrule">custom</span><span class="token punctuation">:</span> <span class="token key atrule">destinationBucket</span><span class="token punctuation">:</span> blender<span class="token punctuation">-</span>on<span class="token punctuation">-</span>lambda<span class="token punctuation">-</span>bucket <span class="token key atrule">provider</span><span class="token punctuation">:</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> aws <span class="token key atrule">region</span><span class="token punctuation">:</span> us<span class="token punctuation">-</span>east<span class="token punctuation">-</span><span class="token number">1</span> <span class="token key atrule">memorySize</span><span class="token punctuation">:</span> <span class="token number">10240</span> <span class="token key atrule">timeout</span><span class="token punctuation">:</span> <span class="token number">900</span> <span class="token key atrule">ecr</span><span class="token punctuation">:</span> <span class="token key atrule">images</span><span class="token punctuation">:</span> <span class="token key atrule">blender-container-image</span><span class="token punctuation">:</span> <span class="token key atrule">path</span><span class="token punctuation">:</span> ./ <span class="token key atrule">iamRoleStatements</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">Effect</span><span class="token punctuation">:</span> <span class="token string">"Allow"</span> <span class="token key atrule">Action</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token string">"s3:PutObject"</span> <span class="token punctuation">-</span> <span class="token string">"s3:GetObject"</span> <span class="token key atrule">Resource</span><span class="token punctuation">:</span> <span class="token key atrule">Fn::Join</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token string">""</span> <span class="token punctuation">-</span> <span class="token punctuation">-</span> <span class="token string">"arn:aws:s3:::"</span> <span class="token punctuation">-</span> <span class="token string">"${self:custom.destinationBucket}"</span> <span class="token punctuation">-</span> <span class="token string">"/*"</span> <span class="token key atrule">functions</span><span class="token punctuation">:</span> <span class="token key atrule">render</span><span class="token punctuation">:</span> <span class="token key atrule">image</span><span class="token punctuation">:</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> blender<span class="token punctuation">-</span>container<span class="token punctuation">-</span>image <span class="token key atrule">maximumRetryAttempts</span><span class="token punctuation">:</span> <span class="token number">0</span> <span class="token key atrule">resources</span><span class="token punctuation">:</span> <span class="token key atrule">Resources</span><span class="token punctuation">:</span> <span class="token key atrule">S3BucketOutputs</span><span class="token punctuation">:</span> <span class="token key atrule">Type</span><span class="token punctuation">:</span> AWS<span class="token punctuation">:</span><span class="token punctuation">:</span>S3<span class="token punctuation">:</span><span class="token punctuation">:</span>Bucket <span class="token key atrule">Properties</span><span class="token punctuation">:</span> <span class="token key atrule">BucketName</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>self<span class="token punctuation">:</span>custom.destinationBucket<span class="token punctuation">}</span></code></pre></div> <p>In this example, I am using the largest available Lambda functions with 10GB or RAM and 6 vCPUs so that the rendering goes as fast as possible.</p> <h3>The Python Lambda handler</h3> <p>The <code>app.py</code> file contains the logic that will be triggered by the Lambda infrastructure and it will receive the event passed as a trigger. In this case, the event must contain the <code>width</code> and <code>height</code> of the image to be rendered.</p> <div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">import</span> json <span class="token keyword">import</span> os <span class="token keyword">def</span> <span class="token function">handler</span><span class="token punctuation">(</span>event<span class="token punctuation">,</span> context<span class="token punctuation">)</span><span class="token punctuation">:</span> os<span class="token punctuation">.</span>system<span class="token punctuation">(</span><span class="token string-interpolation"><span class="token string">f"blender -b -P script.py -- </span><span class="token interpolation"><span class="token punctuation">{</span>event<span class="token punctuation">.</span>get<span class="token punctuation">(</span><span class="token string">'width'</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">}</span></span><span class="token string"> </span><span class="token interpolation"><span class="token punctuation">{</span>event<span class="token punctuation">.</span>get<span class="token punctuation">(</span><span class="token string">'height'</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">}</span></span><span class="token string">"</span></span><span class="token punctuation">)</span> <span class="token keyword">return</span> <span class="token punctuation">{</span> <span class="token string">"statusCode"</span><span class="token punctuation">:</span> <span class="token number">200</span><span class="token punctuation">,</span> <span class="token string">"body"</span><span class="token punctuation">:</span> json<span class="token punctuation">.</span>dumps<span class="token punctuation">(</span><span class="token punctuation">{</span><span class="token string">"message"</span><span class="token punctuation">:</span> <span class="token string">'ok'</span><span class="token punctuation">}</span><span class="token punctuation">)</span> <span class="token punctuation">}</span></code></pre></div> <h3>The Python Blender script</h3> <p>And finally, here is the python script that will be executed from inside Blender to do the actual rendering.</p> <div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">from</span> datetime <span class="token keyword">import</span> datetime <span class="token keyword">import</span> os <span class="token keyword">import</span> sys <span class="token keyword">import</span> boto3 <span class="token keyword">import</span> bpy argv <span class="token operator">=</span> sys<span class="token punctuation">.</span>argv argv <span class="token operator">=</span> argv<span class="token punctuation">[</span>argv<span class="token punctuation">.</span>index<span class="token punctuation">(</span><span class="token string">"--"</span><span class="token punctuation">)</span> <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">:</span><span class="token punctuation">]</span> s3 <span class="token operator">=</span> boto3<span class="token punctuation">.</span>resource<span class="token punctuation">(</span><span class="token string">"s3"</span><span class="token punctuation">)</span> BUCKET_NAME <span class="token operator">=</span> <span class="token string">"jrb-material-renderer-bucket"</span> filename <span class="token operator">=</span> <span class="token string-interpolation"><span class="token string">f"</span><span class="token interpolation"><span class="token punctuation">{</span>datetime<span class="token punctuation">.</span>now<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>strftime<span class="token punctuation">(</span><span class="token string">'%Y_%m_%d-%I:%M:%S_%p'</span><span class="token punctuation">)</span><span class="token punctuation">}</span></span><span class="token string">.png"</span></span> s3<span class="token punctuation">.</span>Bucket<span class="token punctuation">(</span>BUCKET_NAME<span class="token punctuation">)</span><span class="token punctuation">.</span>download_file<span class="token punctuation">(</span><span class="token string">"scenes/scene.blend"</span><span class="token punctuation">,</span> <span class="token string">"/tmp/scene.blend"</span><span class="token punctuation">)</span> bpy<span class="token punctuation">.</span>ops<span class="token punctuation">.</span>wm<span class="token punctuation">.</span>open_mainfile<span class="token punctuation">(</span>filepath<span class="token operator">=</span><span class="token string">"/tmp/scene.blend"</span><span class="token punctuation">,</span> load_ui<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span> bpy<span class="token punctuation">.</span>context<span class="token punctuation">.</span>scene<span class="token punctuation">.</span>render<span class="token punctuation">.</span>filepath <span class="token operator">=</span> <span class="token string-interpolation"><span class="token string">f"/tmp/</span><span class="token interpolation"><span class="token punctuation">{</span>filename<span class="token punctuation">}</span></span><span class="token string">"</span></span> bpy<span class="token punctuation">.</span>context<span class="token punctuation">.</span>scene<span class="token punctuation">.</span>render<span class="token punctuation">.</span>resolution_x <span class="token operator">=</span> <span class="token builtin">int</span><span class="token punctuation">(</span>argv<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span> bpy<span class="token punctuation">.</span>context<span class="token punctuation">.</span>scene<span class="token punctuation">.</span>render<span class="token punctuation">.</span>resolution_y <span class="token operator">=</span> <span class="token builtin">int</span><span class="token punctuation">(</span>argv<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span> bpy<span class="token punctuation">.</span>ops<span class="token punctuation">.</span>render<span class="token punctuation">.</span>render<span class="token punctuation">(</span>write_still <span class="token operator">=</span> <span class="token boolean">True</span><span class="token punctuation">)</span> s3<span class="token punctuation">.</span>Bucket<span class="token punctuation">(</span>BUCKET_NAME<span class="token punctuation">)</span><span class="token punctuation">.</span>upload_file<span class="token punctuation">(</span><span class="token string-interpolation"><span class="token string">f"/tmp/</span><span class="token interpolation"><span class="token punctuation">{</span>filename<span class="token punctuation">}</span></span><span class="token string">"</span></span><span class="token punctuation">,</span> <span class="token string-interpolation"><span class="token string">f"renders/</span><span class="token interpolation"><span class="token punctuation">{</span>filename<span class="token punctuation">}</span></span><span class="token string">"</span></span><span class="token punctuation">)</span></code></pre></div> <h2>Cost</h2> <p>In the us-east-1 region of AWS, the largest available lambda will cost you $0.01 per minute. The request charge (the amount charged for every invocation) is probably negligible in comparison. Also, bear in mind that <a href="https://aws.amazon.com/free/?all-free-tier.sort-by=item.additionalFields.SortRank&#x26;all-free-tier.sort-order=asc&#x26;awsf.Free%20Tier%20Types=*all&#x26;awsf.Free%20Tier%20Categories=*all&#x26;all-free-tier.q=lambda&#x26;all-free-tier.q_operator=AND">the generous free tier of Lambda</a> might cover your costs! Same for S3.</p> <p>I hope this can help anyone facing the same need that I did.</p> <p>If you're interested in offloading your Blender render jobs to the cloud, <a href="https://twitter.com/jiherr">follow me on Twitter</a>. I'll be posting more content in the coming weeks such as:</p> <ul> <li><a href="/2021/11/blender-serverless-cloud-comparison/">cost comparison with using other cloud providers</a></li> <li>the cost impact of using GPU-enabled EC2 instances instead of Lambda</li> <li>leveraging AWS EFS instead of S3 to load the scene into Blender</li> </ul> <p>If there are other topics you'd be interested in, let me know on Twitter! My DMs are open.</p><![CDATA[Sending Financial Data to Salesforce Without the Hassle Thanks to Codat and Amazon AppFlow]]>/2021/08/amazon-appflow-codat-financial-data//2021/08/amazon-appflow-codat-financial-data/Wed, 11 Aug 2021 00:00:00 GMT<p>We’ve recently built a secure data pipeline for a customer called <a href="https://foundersfirstcapitalpartners.com/">Founders First Capital Partners</a>. They needed to automatically load large amounts of financial data into Salesforce, their CRM, in order to make investment decisions. We used Amazon’s managed data integration service — <a href="https://aws.amazon.com/appflow/">AppFlow</a> — which takes care of creating a secure connection and sending data to Salesforce. To retrieve the data securely from several financial tools, we leveraged <a href="https://www.codat.io/">Codat</a>. Codat's single API powers robust integrations to over 29 different financial tools including QuickBooks, Stripe, Plaid and Oracle NetSuite. Codat takes care of all the heavy lifting involved in these integrations — authorization, data standardization,ongoing connectivity — leaving us free to focus on how to best use this data.</p> <h2>The architecture</h2> <p>We designed a simple serverless architecture that fit our needs.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/2c91bf1a1cb39a8d7b8bb7a6c62d269f/50383/architecture-diagram.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 72.43243243243242%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAIAAACgpqunAAAACXBIWXMAAAsSAAALEgHS3X78AAABz0lEQVQoz4VR23LTMBD1r/ItvPYnmAEeeYRpS5lhSktToK3DpImbOHVtR75fFcm6C7kmngylsNJIs6tzzq52Lb1nUvXneaI+uNUivA4bQLiw7anruvpvZu07SvVs+25zNrmJQ0AxK6LbH1eTIAjH1/+Q6zKPIyCkEFLFxYZ0cIswF/xfmf8QJoTBFiqp9PNmPQ0NKp6f3c791ToyBTxLHqBFVYWbiDIqpeiXECYOMSWMS9m7+2evr7VQu8yEUoSwGEt/vLf2O14Fo/v0dxaE0IgRYnIqzVotmdFULSJSF+9fIt9GQnZs2wdhZwi46yTnKZF3DbMOj44iAM4uvi/XQX3xBibLbVj6rw5bJvPjAwqcr4vJ6eqYR+3m7SdD/nx+6TnO63v84ktmXV7dPAShGUacZLPZ3PMDjDAIgflZ/fFApE4ndJbnq7U3nc6iOC2qui9WayqVhTCmlBnJvKjmjsE8mLYY997zvp2e/LSvCSGcCyO6WLpxksrd8NQ4qqGN3KAEH1pSlmWUZMEGUNZLm4ELLgbkAFDjqMbQYAK2xgglmnDighahruvGNCPyd2bGWFU3ddOa3TRtlWYARGmeV3mZroIkL9K86F8fMXJH/gXRxSC/5wodWQAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Architecture diagram of a serverless solution connecting Codat to Salesforce with Amazon AppFlow" title="Architecture diagram of a serverless solution connecting Codat to Salesforce with Amazon AppFlow" src="/static/2c91bf1a1cb39a8d7b8bb7a6c62d269f/50383/architecture-diagram.png" srcset="/static/2c91bf1a1cb39a8d7b8bb7a6c62d269f/1d79a/architecture-diagram.png 185w, /static/2c91bf1a1cb39a8d7b8bb7a6c62d269f/1efb2/architecture-diagram.png 370w, /static/2c91bf1a1cb39a8d7b8bb7a6c62d269f/50383/architecture-diagram.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h2>Benefits of using these tools</h2> <p>Why would you do this instead of “simply” writing the connection to the APIs?</p> <ul> <li>Less code to write. All you have to do is write code that puts data in an S3 bucket and AppFlow will take care of sending it to Salesforce (or any other supported SaaS tool you choose).</li> <li>All the usual benefits of serverless and managed services: you only have to focus on the business logic that matters to your use case. AWS ensures your solution is always up and running (with an SLA of 99.9%). There are no software updates or patching to worry about. It scales with your needs.</li> <li>Enhanced security: for each SaaS that AppFlow connects to, AWS establishes a secure AWS PrivateLink which is a great security guarantee without any additional effort.</li> <li>The standardization of data provided by Codat allowed us to manipulate data from various sources into one consistent data structure, which is a huge time-saver.</li> </ul> <h2>Trade-offs of our solution</h2> <ul> <li>AppFlow can sometimes throw error messages that are not very helpful, which can be frustrating. But, just like with other AWS products, their team is open to feedback and continuously improves AppFlow.</li> <li>The community adoption of AppFlow still seems to be nascent (based on the number of <a href="https://stackoverflow.com/questions/tagged/amazon-appflow">Stack Overflow discussions on the topic</a>), which is a bit of a shame given how helpful this service is. As a result, it can be a struggle to find resources to help you get up and running.</li> </ul> <p>The combination of Codat with Amazon AppFlow allowed us to load large amounts of critical financial data into Salesforce in a very short amount of time, with minimal development effort. I hope you can also save time by using these amazing tools!</p> <hr> <p><em>Thanks to Nate Ritter, CTO of Founders First Capital Partners, for giving us the opportunity to work on this small piece of serverless sorcery. Founders First Capital Partners are based in San Diego, CA and support diverse founder-led small businesses and those in low to moderate income areas by providing them with funding and acceleration services.</em></p><![CDATA[Hasura: GraphQL Without the Baggage.]]>/2021/08/hasura-graphql-without-the-baggage//2021/08/hasura-graphql-without-the-baggage/Wed, 04 Aug 2021 00:00:00 GMT<p><strong>Spoiler Alert:</strong> <em>Star Wars references below. Proceed at your own risk...</em></p> <p>The introduction of GraphQL back in 2015 garnered a lot of hype in the development sphere - allowing clients to describe exactly what data they require (preventing over-fetching unneeded information) and unifying multiple resources into a singular request. This substantially improved scalability and reusability of backend endpoints, and makes the development experience all-in-all more pleasant.</p> <p>So you might be thinking to yourself: <em>"Great. What's the catch?"</em></p> <p>Well... Implementing GraphQL into your backend will inadvertently increase complexity - you'll need to dedicate time on defining types, <em>efficient</em> resolvers, mutators, queries, etc. If you're working as a full-stack developer, you might find that using GraphQL is kind of like taking a dollar from one pocket and putting it in another. The added complexity can start to outweigh the benefits, especially on smaller scale projects. On top of that, caching is <em>complex</em> with GraphQL. With no in-built caching capabilities, this often becomes a pain point for developers.</p> <p>That's not to diminish or underplay the clear benefits of using GraphQL.</p> <p>However, I want to use this opportunity to touch on some of the less talked about vices of GraphQL, and present <a href="https://hasura.io/">Hasura</a> as a possible solution to some of these concerns. Hasura is a GraphQL service that <strong>automatically</strong> generates GraphQL endpoints from your database schemas, taking out a lot of the complexities and manual work needed. Simply by connecting Hasura to your SQL database, it will generate schemas and resolvers using the existing tables and views. It's quite magical in how quick it is to get up and running...</p> <h1>GraphQL Complexities</h1> <h2>Boilerplate &#x26; Initial Setup</h2> <p>As you'll usually find with most other large scale dependencies or libraries, the addition of GraphQL will require some level of initial setup and boilerplate, you're adding a new layer to both your frontend and your backend. You're probably going to need to add a GraphQL client, define the types, schemas, error handling, caching on the frontend level. On the backend side, you'll need to define multiple resolvers to fetch different views of the data (get all the data/get a single datapoint with an id), as well as defining mutations for creating new datapoints, editing existing data, and deleting data.</p> <p>Now you could argue that these would all still be required if you used normal REST endpoints - and you would be right. However, on top of these, you'll need to deal with typing on the backend, caching (you can't rely on native HTTP caching like REST APIs), higher order components, and additional validators to catch some edge-cases that arise from defined schemas.</p> <p>A solution commonly taken on the backend side, is to define boilerplate <em>abstractions</em> to take the repetition out of having to define basic resolvers/queries for new data classes you add into your application, but again this takes time to setup (plus you're adding more complexity). While this might not be an issue for larger projects that can reap the benefits of using GraphQL, it becomes a massive swaying point away for small projects.</p> <h2>Resolver Hell</h2> <p>Resolvers are the functions that resolve the values for any field in a schema. They can return primitive values, or objects that get further broken down into more primitive types. You'll find that these typically return values from databases or other APIs.</p> <p>Now there are a number of databases and ORMs that come with GraphQL mappers that will automatically handle resolvers for you when you are defining your data in objects. For example, in a project that I worked on, we use <a href="https://gorm.grails.org/">GORM</a>, which handles resolving and schema generation. However, these do come with their overheads, and you would need to define your data models according to these data access tools.</p> <p>In many cases you'll find that you're dealing with your database directly, and you'll be doing the DB querying yourself. This opens a whole new can of worms to deal with.</p> <p>A common issue that many face is over-querying the database. Let's demonstrate this with an example:</p> <blockquote> <p>You are creating an application to manage a large movie cinema company with multiple branches. You have a query that fetches a list of all the cinemas, the screens in each, the movies showing on each screen, and which production companies released them.</p> </blockquote> <p>Now presume that there are 5 cinemas (branches), with 10 screens each, each playing 5 movies at any given point. This means that we have 1 query to get the cinemas, 5 queries for each cinema getting the screens, and 50 queries for the movies playing on each screen (50 calls to get the details of the movie playing on each screens). That comes down to a total of 56 database calls for this query.</p> <p>Say you wanted to fetch some basic metadata about each of these movies as well (such as the production companies), that would be another 50 database calls added, which would increase our total to a whopping 106 database calls...</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/08c2ecb6089ea60fb3186baa3aef7a4b/50383/Untitled_Diagram.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 53.51351351351351%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAC4jAAAuIwF4pT92AAABOklEQVQoz31SSY6DQAzk/39AAo58hQA55IiEOARIWEXCvjhTlkwaZjQllaBNu1y20egH27bhQeM40jAMO3Hu+55er9eBTdP8ioHLspAmYuu60jRNdAZi+IZ7wjMkFobhV3CeZ6YKCMHlWSyOY6rrmsqypMfjwd0AruuSplaBm67rdlcooLoWQV3XyXEcLiatAtfrlTSow2qe50zbtulyuVCWZXS/3+n9flMQBFwIjpBsGAb5vr93hjhyEdPatmP7Muwoiuj5fO6O2rZlQRTGggDTNNkNukAcebjned6xZbGuzvCvli3L4nkBag63rC5FkiWGpyxFRVVVPAK1MMAO5TIqQfCcjEIQlb9AisrSkJckCfN2u30FMR8EMUtcViliqqi8Q7AoCs7FAg+CaZrygJEAV0Kc/6OIQ+sDPYRSQ8L0gXgAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Diagram showing number of calls expanding as we go deep into the tree." title="Diagram showing number of calls expanding as we go deep into the tree." src="/static/08c2ecb6089ea60fb3186baa3aef7a4b/50383/Untitled_Diagram.png" srcset="/static/08c2ecb6089ea60fb3186baa3aef7a4b/1d79a/Untitled_Diagram.png 185w, /static/08c2ecb6089ea60fb3186baa3aef7a4b/1efb2/Untitled_Diagram.png 370w, /static/08c2ecb6089ea60fb3186baa3aef7a4b/50383/Untitled_Diagram.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>Cinema example above visualised.</p> <p>This can be mitigated by strategising and coming up with efficient queries and resolvers, but the problem remains that it is <strong>very easy</strong> to fall into these common patterns that will result in overfetching. It's just the price that comes with allowing the client to specify their data needs and schemas.</p> <p>There are solutions to deal with this - for example, <a href="https://github.com/graphql/dataloader">Dataloader</a> will batch some of these database calls together and cache some of the results. However, these solutions are far from perfect and they have limitations when making certain queries. Not to mention that by adding new dependencies, we start inadvertently increasing complexity of the application.</p> <h2>Performance &#x26; Caching</h2> <p>With complex views on the data, comes the cost of query complexity and the number of database calls increasing. It becomes a balancing game... You'll want to use GraphQL's ability to define graph-based traversals to fetch multiple resources (in fact, this is probably the single biggest selling point IMHO), but use it <strong>too</strong> much (or irresponsibly), and you might find that your queries become slow and start to massively affect performance.</p> <p>Take the Star Wars example that is shown on the homepage of the GraphQL website:</p> <div class="gatsby-highlight" data-language="graphql"><pre class="language-graphql"><code class="language-graphql"><span class="token keyword">type</span> <span class="token class-name">Query</span> <span class="token punctuation">{</span> <span class="token attr-name">hero</span><span class="token punctuation">:</span> Character <span class="token punctuation">}</span> <span class="token keyword">type</span> <span class="token class-name">Character</span> <span class="token punctuation">{</span> <span class="token attr-name">name</span><span class="token punctuation">:</span> String <span class="token attr-name">friends</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>Character<span class="token punctuation">]</span> <span class="token attr-name">homeWorld</span><span class="token punctuation">:</span> Planet <span class="token punctuation">}</span> <span class="token keyword">type</span> <span class="token class-name">Planet</span> <span class="token punctuation">{</span> <span class="token attr-name">name</span><span class="token punctuation">:</span> String <span class="token attr-name">climate</span><span class="token punctuation">:</span> String <span class="token punctuation">}</span></code></pre></div> <p>Let's try and visualise it with a flowchart, with the main trio of the original series. If Luke Skywalker is the hero of the original series (which is a false statement, because we all know the Ewoks were the true heroes), he has several friends, and each of these friends have a <code>homeWorld</code>.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/111ce09a46692405edf60c8ca043138c/07f3a/lukeskywalker.jpg" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 121.62162162162163%; position: relative; bottom: 0; left: 0; background-image: url('data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAYABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAECBf/EABQBAQAAAAAAAAAAAAAAAAAAAAD/2gAMAwEAAhADEAAAAe/ENshQyD//xAAXEAADAQAAAAAAAAAAAAAAAAAAASAR/9oACAEBAAEFAhRsf//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQMBAT8BH//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQIBAT8BH//EABQQAQAAAAAAAAAAAAAAAAAAADD/2gAIAQEABj8CH//EABgQAAMBAQAAAAAAAAAAAAAAAAABETEQ/9oACAEBAAE/IRJy0zCRa+Jo/9oADAMBAAIAAwAAABCQBzz/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/EB//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/EB//xAAcEAACAgIDAAAAAAAAAAAAAAABEQAhEEExYaH/2gAIAQEAAT8QPFxQhK/MFiYA3AQDIJ252iG7K1ghsu6n/9k='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Graph showing analogy of Star Wars graph." title="Graph showing analogy of Star Wars graph." src="/static/111ce09a46692405edf60c8ca043138c/07f3a/lukeskywalker.jpg" srcset="/static/111ce09a46692405edf60c8ca043138c/d7fe6/lukeskywalker.jpg 185w, /static/111ce09a46692405edf60c8ca043138c/f4308/lukeskywalker.jpg 370w, /static/111ce09a46692405edf60c8ca043138c/07f3a/lukeskywalker.jpg 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>GraphQL allows you to traverse the graph of Characters by defining resolvers to fetch the friends. Now this is a fairly simple example, and you'll find that real world data will usually have more complex relations and hence this graph will become larger. This will take a toll on performance, and you may find that certain queries will take unreasonably long to run.</p> <p>A common approach taken to solve this problem is through caching, and although it won't work well for frequently changing data, this can help reduce the number of queries that need to be made regularly on data that rarely changes.</p> <p>As an example, Apollo can allow you to define and control caching structures on your web applications, but if you have multiple frontend applications (say a web, Android, and iOS app), you'll probably need to handle caching on each of these levels separately. Again, added complexity that will be introduced by using GraphQL.</p> <p>However, this options is only viable for data that isn't changing regularly... So in the case where Leia's home planet of Alderaan gets obliterated by the Death Star, or Han Solo and Leia become separated (ergo are no longer considered friends) due to their son falling to the dark side of the force, you may find that caching will result in data inaccuracies. This is where the importance of having <strong>well-written and efficient</strong> resolvers and database calls comes into play yet again.</p> <p><em>And with that, all may seem lost, but from the darkness of caching, boilerplate, and complex resolvers comes the saving grace of Hasura...</em></p> <h1>Hasura</h1> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/8e8c2e9917ba8601ac1ff7a927f8b41c/50383/Untitled.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 56.21621621621622%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAAsSAAALEgHS3X78AAABZklEQVQoz41STUsDMRDNP/fm2ZMeLHio+AcED/4E8VBpqYKU9VJaithuu5vububDmXxsU61g4O1sJvNeXjIxiMggwBMAgJN5JMrWYoxzQ/KhKOrgH4Jdx2itF83zygUkNg04n9h2jud1y8yUFcGRe0hOVFTIu0YhhkQ8mBFBJbVRtHUBSlRhqfTAeAolSiYeObgKzkLUGqPFnYj64uhCRT6qhicb6yPJPenGnThTUNokcjA61LmBKJgfS909zNd8v9jycFLw4OaOrwa3fHk95LPzC356HrGOdMfgnN/QN8Xbja6SqP4vbMuPy5KL9Y5ns4Jfxq88fXvn0XjK88XSC6b6ja37UxrxLBPHq70NyUxU7+mvAbFhoQdwcGg7JIUsehChaB6gOedA4HqkvEbS1lKAck2xlgYINnvi/JGr/aqqf73F/t6yt/tliT/rMDdlg1TuFUR1ix5pdyEfuf0JrWsc9jwRp29i/ls8Hl+ptAAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Hasura Logo" title="Hasura Logo" src="/static/8e8c2e9917ba8601ac1ff7a927f8b41c/50383/Untitled.png" srcset="/static/8e8c2e9917ba8601ac1ff7a927f8b41c/1d79a/Untitled.png 185w, /static/8e8c2e9917ba8601ac1ff7a927f8b41c/1efb2/Untitled.png 370w, /static/8e8c2e9917ba8601ac1ff7a927f8b41c/50383/Untitled.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>Hasura is an open source service that allows you to take your relational databases, and automatically generate GraphQL APIs on top of them. Effectively, this removes one of the biggest challenges that backend developers using GraphQL face, which is creating efficient SQL queries and resolvers.</p> <p>Looking at the challenges posed above, the issue of <strong>excessive boilerplate code &#x26; high initial costs</strong> effectively become neutralised by using a tool like Hasura. I had a Postgres database running on my machine, and it took me a total of 15 minutes to spin up Hasura with Docker. Just by changing the <code>docker-compose</code>, I pointed it to my database, and it loaded up already connected and hosting a GraphQL endpoint with my defined views! Honestly, it felt like some form of vodoo magic! The steps I followed were really quite simple:</p> <ul> <li>Fetch the docker-compose file with <code>curl</code>.</li> <li>Run <code>docker-compose up -d</code></li> <li>Get yourself a cup of coffee while you wait for it to run.</li> <li>Open the console page at <code>localhost:8080/console</code></li> <li>Click on the data tab, and click the "Connect Database" button.</li> <li>Add a name for your database, the SQL flavour, and lastly the database URL.</li> <li>Click the connect button.</li> <li>After it's done connecting, click on the "GraphiQL" tab, and see that your database tables can be queried out of the box!</li> </ul> <p>Now, a common question that may come up about the initial cost of using a technology is how well it integrates with existing codebases. Presume that you're already using a GraphQL endpoint for the backend of your application, and you want to incrementally introduce Hasura to handle the basic queries and subscriptions (yes, it comes prebuilt with subscriptions, and that can be an absolute pain with certain GraphQL servers). You can use a feature called <em>Remote Schemas</em> to point Hasura to your already existing GraphQL endpoint, and it'll automatically combine the schemas together to create a unified GraphQL endpoint that your frontend can use. Additionally, you can use this to delegate more complex business logic in your code to other services (such as a payment API).</p> <p>The second challenge touched on above was the issue of <strong>inefficient and complex resolvers</strong>. Without getting into much detail, Hasura compiles down GraphQL queries using abstract syntax trees, validates and checks permissioning at this level, and then constructs the entire query into one raw SQL query. On top of this, they do 2 levels of caching (one at the SQL compilation level, and another at the database level with prepared statements) within their architecture to reduce the amount of times these compilation steps are required. They're building and improving on this architecture and performance with every update. For example, they've introduced JSON aggregations on the database level, effectively removing the need to flatten the results from the database calls in the backend - they claim this increases performance anywhere from 5-8 times! For those who are interested, they have an interesting <a href="https://hasura.io/blog/fast-graphql-execution-with-query-caching-prepared-statements/">article</a> on their website talking about their architecture in more detail.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/b50ab2f1883ef964b11913125ee5e396/50383/DataWrapper.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 114.05405405405406%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAXCAYAAAALHW+jAAAACXBIWXMAAAsSAAALEgHS3X78AAADbklEQVQ4y41ViXLiOBDl/79oq3ZnazOpEDB2AAOBgO8DbPCFbXxgw9uWSCgymWFG1KtuSa1HX5I7oHE+nxGGITRNg+/7+Fj7HT7sbkfnY6FpGqRpirIs8afjR7JPhJv1BpqqwXVc1HWNKIqQJAniOOE6+zMm2ZxFwxy466HrurBMC7Zt8wOyPIEoSpCkFyyXK54OJtlcUVTkeX6f0PM8xFEM3/ORZRksy4KuGxyO4/B907ys2baDqqruE7KQGEkcx1fDpjmiKIvroaIoUB/ru4Xq3LJXZfUlyXVVEzvQNi2O1fG3heIeZnWKuAyxS334yYbrF0SIDgHsrYl16CBh84ohvEGEkGRSJ2hPDTotTlhoPeiLZyjzZ7zK32GuBGivXQ5blTAdPdDaAPpbD/a0B28iYDMlvEtP7sNeiggOWyI8t5ipT6jLA7bU1LPphLu+9hTEgU2ts4dh6LApv8WxQCCLKHoCCkHkOBCqnojtSMA236DTkJsLsw/dXGEov+Cp94iFsoBiTajviDCNsVLmVDAN9blEPB8iEwRkkohMFLk8DET4soBd7hEhhazOu4jEZ0TDAUFA+NJHTLCfHyE8fMNs8Ixh9wHC479Quk8oiSAXBuTdBWV/gN14QB5ywhbq4kIWT4aIyUuGPen+WILw3z8Ydb9jQMRPf/+FN7GHZDZEOJEILxzM3pn1Lx6yHMrKI9IyQXmqcGgOqOmnWCokWYJuKUgPMabzMSavY8ivIyz1Jcq2QNFcULYlVoaE3X6NzolCXlkSJX4I057AtOQrNH0IlWA5U7xRledvfcqnCN0Y3djRGXNMNRhRi+0o5PYIO9YQFQFfYIhJd6nCpqdRDxq8D+2dST26huXrtGe92wbv9iEPN8ypbeqmghPrXzqe3Wn2+rB76xBMw8Ruu8N8vsDaXX+x3xf0DqQu5ZDaRgsWnNRNjAtiA5vUoqq52B5c+JnDdS5pvkntqx0H6ex8yBr72NZ8sW4rMG9vkRcp78O8yKgwe+zzBNkhpb3yalORztLGwg4y79ch8zDolsxmM3q+XCiqihW9hywFPxtp+R7y6dTCihRO6sTGJ7lOqBAU4jpQsd4p8HKXp+Kzrc4jtCMNWZVcni9Gytz+EcemxpEav4wCFFsP7FaxtZ/Zsiv86YG9+zGqjzhX1R99tP4HWtfHwrVLFw0AAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Hasura&#39;s Data Wrapper" title="Hasura&#39;s Data Wrapper" src="/static/b50ab2f1883ef964b11913125ee5e396/50383/DataWrapper.png" srcset="/static/b50ab2f1883ef964b11913125ee5e396/1d79a/DataWrapper.png 185w, /static/b50ab2f1883ef964b11913125ee5e396/1efb2/DataWrapper.png 370w, /static/b50ab2f1883ef964b11913125ee5e396/50383/DataWrapper.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>Hasura's Data Wrapper (Source: hasura.io)</p> <p>Lastly, we touched on the issue of <strong>caching &#x26; performance</strong>. Within the data wrapper layer, there's already 2 levels of caching that will <em>significantly</em> improve performance. Additionally, there is another layer of caching, which is on the query level, that effectively allows you to cache the response being sent back to the client for a defined period of time. You can make a call with the <code>@cached</code> directive to cache the response, supplying an optional <code>ttl</code> argument (how long to cache the response):</p> <div class="gatsby-highlight" data-language="graphql"><pre class="language-graphql"><code class="language-graphql"><span class="token keyword">query</span> MyCachedQuery <span class="token directive function">@cached</span><span class="token punctuation">(</span><span class="token attr-name">ttl</span><span class="token punctuation">:</span> <span class="token number">120</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> users <span class="token punctuation">{</span> id name <span class="token punctuation">}</span> <span class="token punctuation">}</span></code></pre></div> <p>This caching comes working out of the box. In contrast to common caching approaches (such as using Apollo), having this caching done on the server side can be beneficial for unifying the responses on any given platform at any point of time. Imagine you have a cached query for the top players in a leaderboard. If the caches are set at different points on the client, there may be inconsistencies in how this data is displayed. Caching on the server-side circumvents this.</p> <p>On top of that, Hasura's performance isn't only due to compiled queries, efficient resolvers and caching. There's a ton more work put into improving speed and performance and it is a regularly improved and maintained project that's already heavily utilised in the industry.</p> <p>So to demonstrate the Star Wars example we saw above, I went ahead and created a database schema to have the following tables:</p> <ul> <li>Character</li> <li>Planet</li> <li>friendsWith</li> </ul> <p>I did all of this through the Hasura data admin panel (sort of like a lite version of PGAdmin). Through this data panel, you can choose which tables/views can be tracked by the GraphQL schema, and you can define table relationships and how they manifest in the GraphQL schema.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/60a66d7fe305da285859ae536b2f6d5d/50383/character-table.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 46.48648648648649%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAYAAAAywQxIAAAACXBIWXMAABYlAAAWJQFJUiTwAAABs0lEQVQoz21R227UMBDN90Gp6FIKpVJb2tUCT4jyc3wBTzzTB9SC2O0mcRI78S12DHsYz7alQlg6mpEv5zIuTucf8HrxETuzczx+eoK9FwvMXr7B4asjzA7OsPt8gd39OZ4dvsXewYLrzuwMx/MLHJ2+x6P9czyh/e3ZOxTljy+oll9R3nxHV17i+uoS366uUYoKN3WLZdVhua6xrgSaTjFEK1E3EiWdV6JjcN90KNIUYa2GNRoxBnhv0DQr3hNNA6kkQXHftC3atkPbdUi/N/9F0WuLQTsi8HDeozce2ie4MWAw9i+0gabaSoV+0BhDhGeE27pFEb1FIld3y+oaWv6EdR7GWsQp8aMp/WKMcSLhkfs4TUyc7+Tek4miVz0UqVpryFWCWX+GWX6C7A1H1dbBOEpAJBmGhLTNrg0TK3Kb0wQizWKFowuBHIzjyLBupFlJDMMAQ48UkUopsVqtUFUVhBAQ9DGKZtupAbVo4EjwnhD/LE+kdV3TnAZksZgjUs2kWmsWMT4gpURORwQ65+hUM2mx2WzwENuZRVLvOW6+mOcZpu3s7K2bHFP2Pe/ducv4AxvzkIE5+FdjAAAAAElFTkSuQmCC'); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Hasura Data Admin Panel" title="Hasura Data Admin Panel" src="/static/60a66d7fe305da285859ae536b2f6d5d/50383/character-table.png" srcset="/static/60a66d7fe305da285859ae536b2f6d5d/1d79a/character-table.png 185w, /static/60a66d7fe305da285859ae536b2f6d5d/1efb2/character-table.png 370w, /static/60a66d7fe305da285859ae536b2f6d5d/50383/character-table.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>Switching to the "GraphiQL" tab, we can now see that the explorer now has a bunch of different datapoints:</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 458px; " > <a class="gatsby-resp-image-link" href="/static/0389299140b70432fc0056b3a9915721/f7a31/explorer.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 74.5945945945946%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAPCAYAAADkmO9VAAAACXBIWXMAABYlAAAWJQFJUiTwAAABjUlEQVQ4y5WU626CMBiGvf+Lmz90iWbztKEIAqVHDi2+61eHERfRkTSEQB/eQ9tJW9doSwZrNGyp0GxitLsELmE4Wwe6zuczxq7+vXMOk0ZwqMUMdbSFzQuw6RL52wJsvoKzdjDhEXgA1Foj0xZxWePANHTXjU56CpRCIC0kdonA5sihqxZt62DqFlVj/d2i6/6hUCmJpFAB9n2SSEuD6SrF2j/Ptxk+9gx1614HSimQMIWtV0iDgJEH0w+Eaa4TxnL8o7CUFTYets9UUHnIFbS33Ft9luOwFKVQCBPUxcUFmIvq+iFB+/GSQiol5zpkFmUSX6nAiRus4hIzn+EyYiHTo18BrwElWTbYpZSlDnYzr3Dv7/QDiuF+4j10mKGklqVXxIPKdXxpe5vwYL1U9dPFPQAKv1OYV3jiVbC7+bVObT9SNqqQdgoBKaMeSDbpmXJ83+X4PFzyzG7KegjknCPzh0Jc6AAjSL90qHlSS6PxB4V13XOgMSZsN+az6kuhlik/sp2UOrwbW4u3wB+cnZBXc03oKAAAAABJRU5ErkJggg=='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Explorer for the Star Wars example" title="Explorer for the Star Wars example" src="/static/0389299140b70432fc0056b3a9915721/f7a31/explorer.png" srcset="/static/0389299140b70432fc0056b3a9915721/1d79a/explorer.png 185w, /static/0389299140b70432fc0056b3a9915721/1efb2/explorer.png 370w, /static/0389299140b70432fc0056b3a9915721/f7a31/explorer.png 458w" sizes="(max-width: 458px) 100vw, 458px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>You can fetch the list of each of the data types by calling the basic data type (for example, <code>Planet</code>), and this comes with pagination and filtering out of the box. Alternatively, if you're looking for some form of aggregation on the data, you can use the <code>_aggregate</code> datapoint, which can run some basic aggregation on the data (such as averaging a field, counting, finding the min/max the field, or even getting the standard deviation). Lastly, you can get a single data entry by using the <code>by_pk</code> variation.</p> <p>Now, to see this in action, we construct a query asking for the first character in our <code>character</code> table. We ask for the name of the character, their home planet (along with its associated climate), and a list of their friends.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/851c4ca40221a5ce567a62e4d709bc60/50383/starwars-query.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 72.43243243243242%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAABlElEQVQ4y41Tia7jIBDj/79yr24ucnEnIfEa0r5H21RaJGuABOMZD6KtK1R1i7qVaDuJYeig1IxZmy+oO8q9WWlMs2Y0GCeV98KyQljrYAi/LPBhgfMLli1iixFrgbjv3Nu/1tsdcT943lJUDU0eMVNNUqZ5g3Me1jmcl1jGE2kuZU81qvhWRGOZlWYWGiJuLh/y3mM/jqxkv8BJZjk/8jq+4ODZwCxF1XS4/a0g+x6dlOioNqlJMa2bpj3372i7Lsdxmp4g+wFymCAmLqpfFfpqoMqAdV2I9QlJveTFfT9CGxpkvo16hfDeQc4Oswn4NPY98meLnw2V1CNsN7PegQa+gy4bDNrjJjVYiFyLEmls20YCh98ts7n1sKOGDYEG+jcIxWKn/vHLlg9/Ikzuy0njB0ujSGxnmxU9iB5zoWl1cu1BVo4nQrZHauJ2UNCENe6aMBnxreZZ2WvKIwn/NCMUYfUHQkcHS4JS5WMe+SJS42s2sA9n8W0yoSAqFPo3RVcpp1dQKsq4Tvn/CFMNX9VcKfwH6+g//lFECuYAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="Hasura Data Admin Panel" title="Hasura Data Admin Panel" src="/static/851c4ca40221a5ce567a62e4d709bc60/50383/starwars-query.png" srcset="/static/851c4ca40221a5ce567a62e4d709bc60/1d79a/starwars-query.png 185w, /static/851c4ca40221a5ce567a62e4d709bc60/1efb2/starwars-query.png 370w, /static/851c4ca40221a5ce567a62e4d709bc60/50383/starwars-query.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <p>To be able to do this in the matter of a couple minutes is something that I still find magical.</p> <p>I'll end here with some of the selling points that I think make Hasura really shine:</p> <ul> <li>Automatically generates the schemas and resolvers for your GraphQL endpoints (less pain).</li> <li>Comes prebuilt with option variables that you can use in your queries (for example, <code>limit</code> and <code>offset</code>, which will allow you to more or less handle lazy loading of datapoints).</li> <li>Currently works with a number of SQL databases (Postgres, Microsoft SQL, Amazon Aurora, and Google Big Query) with more coming soon.</li> <li>Easy to integrate with authentication and permissioning platforms, such as Auth0.</li> <li>Gives you a web UI to manage the GraphQL endpoint, as well as an interface to manage the database and its views itself (think PGAdmin).</li> <li>Using Remote Schemas, you can hook Hasura up with other GraphQL endpoints and other services to unify your endpoints.</li> <li>Using <a href="https://hasura.io/docs/latest/graphql/core/actions/index.html">Actions</a>, you'll be able to define calls to custom handlers that will execute more complex business logic (which opens the door to serverless by using Lambdas)</li> <li>You can define event triggers to call actions automatically (when a user signs up, send an email via a Lambda function).</li> <li>Highly maintained and used by the community, with over 23k stars on Github and 250m+ downloads.</li> </ul> <h3>Conclusion</h3> <p>GraphQL provides some fascinating functionality that aims to make the data layer of applications unified and flexible. As with most things, this comes at a cost. The initial setup and boilerplates will mean that you will need to spend time upfront in creating the right infrastructure for using GraphQL. On top of that, the flexibility in schema means that you will need to have additional considerations when it comes to performance, and you will need to come up with efficient queries and resolvers. Lastly, with GraphQL we lose the native HTTP caching capabilities, meaning that caching is not standardized and becomes cumbersome additional work.</p> <p>Hasura, acting as a service layer between your databases and clients, can automatically generate performant GraphQL queries, mutators, and even subscriptions, with minimal effort on the your end.</p> <p>Whilst Hasura's solution to automatic generation of schemas and endpoints might not offer the full flexibility that large-scale highly complex applications might need, it opens the door to using GraphQL on smaller scale projects where the startup costs of introducing it would outweigh the benefits. I believe this is where Hasura will excel, and where it will ultimately find its place in the GraphQL market.</p><![CDATA[Writing a native Ionic plugin for Capacitor in less than 30 minutes]]>/2021/07/writing-a-native-ionic-plugin-for-capacitor//2021/07/writing-a-native-ionic-plugin-for-capacitor/Mon, 12 Jul 2021 00:00:00 GMT<p><strong>TLDR:</strong> <em>When the pandemic first started, I decided to develop a contact-tracing mobile app. I was studying Computer Science at the time and decided to use the same cross-platform framework that we were using on our course - Ionic. I wanted to use Google's <a href="https://developers.google.com/nearby/messages/overview">Nearby Messages API</a> to share packets of information between iOS and Android devices via Bluetooth, but I couldn't find a plugin for this, so I decided to write my own!</em></p> <p><strong>View commits: <a href="https://github.com/JamesDHW/IonicNativePlugin/commit/614b47fe1e54fe43540d3f6b98f0899949673be4">Android plugin</a> → <a href="https://github.com/JamesDHW/IonicNativePlugin/commit/9bcffc1b029ed0422c136fc367e544b4585f9115">iOS plugin</a> → <a href="https://github.com/JamesDHW/IonicNativePlugin/commit/63510c8cc83b570fb639de6573a28bad58b87a3f">Import native plugins</a> → <a href="https://github.com/JamesDHW/IonicNativePlugin/commit/3652b264b287b5c909633e16532225e4293763cb">Implementation</a></strong></p> <p><a href="https://ionicframework.com/">Ionic</a> is a cross-platform mobile framework which allows you to develop an app using JavaScript/HTML/CSS and share this single implementation across different native devices. This Ionic project can then be compiled to native source code (Android/iOS) using Ionic's tool, <a href="https://capacitorjs.com/">Capacitor</a>.</p> <p>Using a cross-platform framework like this to develop your mobile app is almost always a good idea, because it means you can maintain a single codebase which runs on both Android and iOS. This means you're not duplicating business logic across two different codebases in two different languages with two different teams (to be avoided).</p> <p>If you're looking to write some native code in a cross-platform project but you're still deciding which framework to use, check out this <a href="https://blog.theodo.com/2020/04/react-native-bridge-module/">Theodo blog post</a> on React Native as well!</p> <h2>Step 1 - Search for existing solutions</h2> <p>Firstly, let's have a quick check that one of the <a href="https://capacitorjs.com/docs/apis">default plugins</a> doesn't do the job for us already.</p> <p>Secondly, let's search through the <a href="https://github.com/capacitor-community">community plugins</a> to find the feature we're looking for.</p> <p>No luck finding an existing plugin? Don't worry - let's move on to step 2!</p> <h2>Step 2 - Start a new app with Ionic + Capacitor</h2> <p>I'm starting a new project from scratch just to show you how it's done.</p> <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">npm install -g @ionic/cli ionic start &lt;app-name&gt; cd &lt;app-name&gt; ionic serve &lt;-- you can serve your project on localhost</code></pre></div> <p>The <code>/src</code> directory of our Ionic project holds the source code for our cross-platform app.</p> <p>We can use Capacitor to compile this into two native projects:</p> <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">npm install @capacitor/cli @capacitor/core npx cap init npx cap add ios npx cap add android</code></pre></div> <p>When we make changes to our <code>/src</code> directory, we can apply those changes to our native projects by running <code>npx ionic build &#x26;&#x26; npx cap copy</code>.</p> <p>We now have our project set up and ready to add a custom plugin.</p> <h2>Step 3 - Check the documentation</h2> <p>Capacitor is well-documented and it's worth linking some references here to accompany this guide:</p> <p><a href="https://capacitorjs.com/docs/android/custom-code">Getting started with Android</a></p> <p><a href="https://capacitorjs.com/docs/plugins/android">Android reference</a></p> <p><a href="https://capacitorjs.com/docs/ios/custom-code">Getting started with iOS</a></p> <p><a href="https://capacitorjs.com/docs/plugins/ios">iOS reference</a></p> <h2>Step 4 - Define a strategy</h2> <p>There are three main steps to writing a plugin:</p> <ol> <li>(iOS &#x26; Android) Write a class in each native project to hold the native code we want to run (this is our plugin).</li> <li>(iOS &#x26; Android) Register these classes inside the bridge of our native projects to expose the methods.</li> <li>(Javascript - Ionic) Import and call the methods of our registered plugin.</li> </ol> <p>We should avoid duplicating code as much as much as possible (DRY), so all your business logic stays written in JavaScript, keeping the bare minimum written natively.</p> <p>Your plugins should be narrow in scope (do one specific thing) - make sure to follow this philosophy and give your plugin a specific name ('Plugin' is a bad name!).</p> <p>We have two main strategies for calling our plugins:</p> <ul> <li>An (asynchronous) method: we call some native method which we can <code>await</code> in our business logic in Ionic.</li> <li>Event listeners: define a listener in our business logic. Events can be triggered in our native code to pass payloads back to the business logic.</li> </ul> <p>If we are subscribing to a stream of events (expecting our method to resolve with multiple payloads over a longer time period e.g. geolocation/ Bluetooth updates), then event listeners are the appropriate way to handle our native calls.</p> <h2>Step 5 - Write and register the Android plugin</h2> <p>[<a href="https://github.com/JamesDHW/IonicNativePlugin/commit/614b47fe1e54fe43540d3f6b98f0899949673be4">View the commit</a>]</p> <p>Open the <code>/android</code> directory of your project in Android Studio.</p> <p>In <code>/app/src/main/java/../../..</code> (next to <code>MainActivity.java</code>), we can create a new file (I'm calling mine <code>IonicNativePluginExample.java</code>) and write a class which extends the Capacitor <code>Plugin</code> class:</p> <div class="gatsby-highlight" data-language="java"><pre class="language-java"><code class="language-java"><span class="token keyword">package</span> <span class="token comment">/* &lt;package-name> */</span><span class="token punctuation">;</span> <span class="token comment">// e.g. io.ionic.starter</span> <span class="token keyword">import</span> <span class="token namespace">com<span class="token punctuation">.</span>getcapacitor<span class="token punctuation">.</span></span><span class="token class-name">JSObject</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token namespace">com<span class="token punctuation">.</span>getcapacitor<span class="token punctuation">.</span></span><span class="token class-name">Plugin</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token namespace">com<span class="token punctuation">.</span>getcapacitor<span class="token punctuation">.</span></span><span class="token class-name">PluginCall</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token namespace">com<span class="token punctuation">.</span>getcapacitor<span class="token punctuation">.</span></span><span class="token class-name">PluginMethod</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token namespace">com<span class="token punctuation">.</span>getcapacitor<span class="token punctuation">.</span>annotation<span class="token punctuation">.</span></span><span class="token class-name">CapacitorPlugin</span><span class="token punctuation">;</span> <span class="token annotation punctuation">@CapacitorPlugin</span><span class="token punctuation">(</span>name <span class="token operator">=</span> <span class="token string">"IonicNativePluginExample"</span><span class="token punctuation">)</span> <span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">IonicNativePluginExample</span> <span class="token keyword">extends</span> <span class="token class-name">Plugin</span> <span class="token punctuation">{</span> <span class="token annotation punctuation">@PluginMethod</span> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token class-name">NativeMethod</span><span class="token punctuation">(</span><span class="token class-name">PluginCall</span> call<span class="token punctuation">)</span><span class="token punctuation">{</span> <span class="token class-name">JSObject</span> result <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">JSObject</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> result<span class="token punctuation">.</span><span class="token function">put</span><span class="token punctuation">(</span><span class="token string">"message"</span><span class="token punctuation">,</span> <span class="token string">"Hello Android user!"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> call<span class="token punctuation">.</span><span class="token function">resolve</span><span class="token punctuation">(</span>result<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token annotation punctuation">@PluginMethod</span> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token class-name">NotifyListeners</span><span class="token punctuation">(</span><span class="token class-name">PluginCall</span> call<span class="token punctuation">)</span><span class="token punctuation">{</span> <span class="token class-name">JSObject</span> result <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">JSObject</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> result<span class="token punctuation">.</span><span class="token function">put</span><span class="token punctuation">(</span><span class="token string">"message"</span><span class="token punctuation">,</span> <span class="token string">"Hello Android user!"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">notifyListeners</span><span class="token punctuation">(</span><span class="token string">"EVENT_LISTENER_NAME"</span><span class="token punctuation">,</span> result<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span></code></pre></div> <p>Next, we can register this plugin in <code>MainActivity.java</code> where Capacitor initialises its bridge:</p> <div class="gatsby-highlight" data-language="java"><pre class="language-java"><code class="language-java"><span class="token keyword">package</span> <span class="token comment">/* &lt;package-name> */</span><span class="token punctuation">;</span> <span class="token comment">// e.g. io.ionic.starter</span> <span class="token keyword">import</span> <span class="token namespace">android<span class="token punctuation">.</span>os<span class="token punctuation">.</span></span><span class="token class-name">Bundle</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token namespace">com<span class="token punctuation">.</span>getcapacitor<span class="token punctuation">.</span></span><span class="token class-name">BridgeActivity</span><span class="token punctuation">;</span> <span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">MainActivity</span> <span class="token keyword">extends</span> <span class="token class-name">BridgeActivity</span> <span class="token punctuation">{</span> <span class="token annotation punctuation">@Override</span> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">onCreate</span><span class="token punctuation">(</span><span class="token class-name">Bundle</span> savedInstanceState<span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">super</span><span class="token punctuation">.</span><span class="token function">onCreate</span><span class="token punctuation">(</span>savedInstanceState<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">registerPlugin</span><span class="token punctuation">(</span><span class="token class-name">IonicNativePluginExample</span><span class="token punctuation">.</span><span class="token keyword">class</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span></code></pre></div> <p>We can run our project like any other native Android project, by pressing the 'play' button in Android Studio.</p> <p>If you're looking to develop in Kotlin - Android Studio comes with tooling to convert this Java class to Kotlin (right-click the file you want to convert).</p> <p><strong>Note: you might need to edit the Gradle <code>distributionUrl</code> to get the project build to succeed - there's an example in my repo <a href="https://github.com/JamesDHW/IonicNativePlugin/commit/e7abb66621ca0c0444cee55aa7f3d40a621eeef7">here</a>.</strong></p> <h2>Step 6 - Write and register the iOS plugin</h2> <p>[<a href="https://github.com/JamesDHW/IonicNativePlugin/commit/9bcffc1b029ed0422c136fc367e544b4585f9115">View the commit</a>]</p> <p>Open the <code>ios/App</code> directory of your project in Xcode.</p> <p>In <code>/App</code> (next to <code>AppDelegate.swift</code>), we can right-click on the directory to create a new swift file (I'm calling mine <code>IonicNativePluginExample.swift</code>). This will hold our plugin class with our <code>IonicNativePluginExample</code> method.</p> <div class="gatsby-highlight" data-language="swift"><pre class="language-swift"><code class="language-swift"><span class="token keyword">import</span> <span class="token builtin">Foundation</span> <span class="token keyword">import</span> <span class="token builtin">Capacitor</span> @<span class="token function">objc</span><span class="token punctuation">(</span><span class="token builtin">IonicNativePluginExample</span><span class="token punctuation">)</span> <span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">IonicNativePluginExample</span><span class="token punctuation">:</span> <span class="token builtin">CAPPlugin</span> <span class="token punctuation">{</span> <span class="token atrule">@objc</span> <span class="token keyword">func</span> <span class="token function">NativeMethod</span><span class="token punctuation">(</span><span class="token number">_</span> call<span class="token punctuation">:</span> <span class="token builtin">CAPPluginCall</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> call<span class="token punctuation">.</span><span class="token function">resolve</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">"message"</span><span class="token punctuation">:</span> <span class="token string">"Hello iOS user!"</span><span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token punctuation">}</span> <span class="token atrule">@objc</span> <span class="token keyword">func</span> <span class="token function">NotifyListeners</span><span class="token punctuation">(</span><span class="token number">_</span> call<span class="token punctuation">:</span> <span class="token builtin">CAPPluginCall</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">self</span><span class="token punctuation">.</span><span class="token function">notifyListeners</span><span class="token punctuation">(</span> <span class="token string">"EVENT_LISTENER_NAME"</span><span class="token punctuation">,</span> data<span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">"message"</span><span class="token punctuation">:</span> <span class="token string">"Hello iOS user!"</span><span class="token punctuation">]</span> <span class="token punctuation">)</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span></code></pre></div> <p>Next, we must register our plugin in a new Objective-C file called <code>IonicNativePluginExample.m</code> (same name as our plugin file but with a <code>.m</code> extension). When prompted by Xcode, create a Bridging Header file (which is an empty file called <code>App-Bridging-Header.h</code>), then register the plugin like so:</p> <div class="gatsby-highlight" data-language="objectivec"><pre class="language-objectivec"><code class="language-objectivec"><span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">import</span> <span class="token expression"><span class="token operator">&lt;</span>Foundation<span class="token operator">/</span>Foundation<span class="token punctuation">.</span>h<span class="token operator">></span></span></span> <span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">import</span> <span class="token expression"><span class="token operator">&lt;</span>Capacitor<span class="token operator">/</span>Capacitor<span class="token punctuation">.</span>h<span class="token operator">></span></span></span> <span class="token function">CAP_PLUGIN</span><span class="token punctuation">(</span>IonicNativePluginExample<span class="token punctuation">,</span> <span class="token string">"IonicNativePluginExample"</span><span class="token punctuation">,</span> <span class="token function">CAP_PLUGIN_METHOD</span><span class="token punctuation">(</span>NativeMethod<span class="token punctuation">,</span> CAPPluginReturnPromise<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">CAP_PLUGIN_METHOD</span><span class="token punctuation">(</span>NotifyListeners<span class="token punctuation">,</span> CAPPluginReturnPromise<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">)</span></code></pre></div> <p>We can run our project like any other native iOS project, by pressing the 'play' button in Xcode.</p> <h2>Step 7 - Import and call our plugin in Ionic</h2> <p>[<a href="https://github.com/JamesDHW/IonicNativePlugin/commit/63510c8cc83b570fb639de6573a28bad58b87a3f">View the commit</a>]</p> <p>Finally, we can now access the plugins we have written from our Ionic project. I've created a <code>/plugins</code> directory to export mine from. We import <code>registerPlugin</code> from Capacitor to find the plugin we have registered in Android and iOS:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">import</span> <span class="token punctuation">{</span> registerPlugin<span class="token punctuation">,</span> Plugin <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"@capacitor/core"</span><span class="token punctuation">;</span> <span class="token comment">// we can take advantage of TypeScript here!</span> <span class="token keyword">interface</span> <span class="token class-name">NativePluginInterface</span> <span class="token keyword">extends</span> <span class="token class-name">Plugin</span> <span class="token punctuation">{</span> <span class="token function-variable function">NativeMethod</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> Promise<span class="token operator">&lt;</span>Record<span class="token operator">&lt;</span><span class="token string">"message"</span><span class="token punctuation">,</span> string<span class="token operator">>></span><span class="token punctuation">;</span> <span class="token function-variable function">NotifyListeners</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> Promise<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>void</span><span class="token punctuation">></span></span><span class="token plain-text">; }; // it's important that both Android and iOS plugins have the same name export const IonicNativePluginExample = registerPlugin</span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span><span class="token class-name">NativePluginInterface</span></span><span class="token punctuation">></span></span><span class="token plain-text">( "IonicNativePluginExample" );</span></code></pre></div> <p>Now we can call the methods in our plugin and access the native code:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">import</span> <span class="token punctuation">{</span> IonicNativePluginExample <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'./plugins/IonicNativePluginExample'</span> <span class="token operator">...</span> <span class="token comment">// add a listener to native events which invokes some callback</span> IonicNativePluginExample<span class="token punctuation">.</span><span class="token function">addListener</span><span class="token punctuation">(</span><span class="token string">"EVENT_LISTENER_NAME"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> message <span class="token punctuation">}</span></span><span class="token punctuation">)</span> <span class="token operator">=></span> console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>message<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// destructure the methods to call our native code from our non-native app</span> <span class="token keyword">const</span> <span class="token punctuation">{</span> NativeMethod<span class="token punctuation">,</span> NotifyListeners <span class="token punctuation">}</span> <span class="token operator">=</span> IonicNativePluginExample<span class="token punctuation">;</span> <span class="token comment">// native methods are asynchronous</span> <span class="token keyword">const</span> <span class="token punctuation">{</span> message <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">NativeMethod</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// this method will trigger our event listener</span> <span class="token function">NotifyListeners</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre></div> <p><strong>Note: to see our changes reflected when we run the project natively, we need to run <code>npx ionic build &#x26;&#x26; npx cap copy</code>.</strong></p> <p>We now have a cross-platform app set up ready to write a native implementation. We run the same business logic from our Ionic app and get a different implementation for each device we run on.</p> <p>I made a small edit to our Ionic project to create the simple implementation below - you can view the commit <a href="https://github.com/JamesDHW/IonicNativePlugin/commit/3652b264b287b5c909633e16532225e4293763cb">here</a>.</p> <p><span class="gatsby-resp-image-wrapper" style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 740px; " > <a class="gatsby-resp-image-link" href="/static/4518537d6c25b12d55e5bc333d933094/50383/demo-app.png" style="display: block" target="_blank" rel="noopener" > <span class="gatsby-resp-image-background-image" style="padding-bottom: 102.16216216216216%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAABYlAAAWJQFJUiTwAAADJklEQVQ4y5WUS0tbQRTHb1d+AEUU3LhTwSdC3STFVupCUh+gYhcG487YlZ+gLuoiQnxWqJ+h+gFEmiaVaKDUUJqkJtG8zfthcqN5mH/nTB6aNiod+DOHuXN/c+acM0fQarWQSqXo7u5Gb28venp6uMimNYlEAp1Oh0wmA4/bi0AgiHgsgWgkhmg0zhTjdiQcRfIqCWFubg6CIDwq2hMOhfH9hxE/TWbYzi/w5asOGu03WM6s+GX5jTPbOWxWO4SJiQnU1dWhsbERzc3NaGhoQH19PZqamtDS0gLhmYDp6WkYTgxQr61BrVbj/fIy14eVFahUKqhWV/FpdxeGY0MRaDabkc1mkc/nkUwmkUgkkMvlQEOj0UC5sIArthaLxRAKhRAOhyGKIv9eKBRwe3vL7cBlAIJcLofJZOKbaLPf7+fAQCDAN7lcLiwwIMXn70EwrgrQD2F8fByUmKOjI5yenkKv18NoNPKZTrbZbHg7M8MSEeM/0VoFdE80fB4vhNHRUQSDQTw0CCifnWXAeLVXNYABP7uyTCaDx+PhCxRD8oBUjiEBF5WLSJWuXA1CFTDoDxav7PV6H/TQarUyoBKpZKocONQIZgnIPJyfn4fT6eQJodnn8/GZvA5HIrDb7RwoloCUaaoKh8MJi9UJM9NVUrxLikKh4JmMx+MccslOcXijsLmiCMUzDFi8cjolVsJCJZbN5pDOFiBmgFy+6GEoGLoDPnrlRQYU0w/sKJQERNhrqgLeLwnypBbwsSzXBJZH2a4FrFXc/+XhOwakpFApUfxorlLp2UbCkSLQ7Xb/c3rZpjpUsiyjgCdHmr1vDnQ4HJUMllUubIvFgqWlJd5tVllX+bi9jfX1dWxtbWFzcxMbGxvY2dnhXehEfwxhcnISra2t6OvrQ1dXF5/v29TCFKwf7n3ew/OBAchkbzA4+BIjIyN4PTyMV0NDGBsbg0T6Avt7+xCmpqaebLB06HX6Bqn0NdMN0jcZpK5vIJLIZuvpTJa9Zfb0Dg4O0N/fj7a2NnR0dKCzs5OL7Pb2dv7t8PCQPz0768pOh6soZ0nMdpAunEiwBvIH8E8hMKow//EAAAAASUVORK5CYII='); background-size: cover; display: block;" ></span> <img class="gatsby-resp-image-image" alt="demo-app.png" title="demo-app.png" src="/static/4518537d6c25b12d55e5bc333d933094/50383/demo-app.png" srcset="/static/4518537d6c25b12d55e5bc333d933094/1d79a/demo-app.png 185w, /static/4518537d6c25b12d55e5bc333d933094/1efb2/demo-app.png 370w, /static/4518537d6c25b12d55e5bc333d933094/50383/demo-app.png 740w" sizes="(max-width: 740px) 100vw, 740px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" /> </a> </span></p> <h2>More complex plugins</h2> <p>With the above setup, you should be ready to write more complex native code to suit your needs for a cross-platform Ionic application.</p> <p>I wrote a native Plugin for Google's Nearby Messages API, which is a publish-subscribe API made by Google which facilitates the transfer of information between internet-connected Android and iOS devices.</p> <p>Nearby Messages uses a combination of Bluetooth, Bluetooth Low Energy, Wi-Fi, and near-ultrasonic communication between nearby devices to create a unique pairing. This pairing is then used to send small payloads over the internet between nearby devices.</p> <p>Here's a quick code snippet from the Android plugin:</p> <div class="gatsby-highlight" data-language="java"><pre class="language-java"><code class="language-java"><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span> <span class="token keyword">import</span> <span class="token namespace">com<span class="token punctuation">.</span>google<span class="token punctuation">.</span>android<span class="token punctuation">.</span>gms<span class="token punctuation">.</span>nearby<span class="token punctuation">.</span></span><span class="token class-name">Nearby</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token namespace">com<span class="token punctuation">.</span>google<span class="token punctuation">.</span>android<span class="token punctuation">.</span>gms<span class="token punctuation">.</span>nearby<span class="token punctuation">.</span>messages<span class="token punctuation">.</span></span><span class="token class-name">Message</span><span class="token punctuation">;</span> <span class="token keyword">import</span> <span class="token namespace">com<span class="token punctuation">.</span>google<span class="token punctuation">.</span>android<span class="token punctuation">.</span>gms<span class="token punctuation">.</span>nearby<span class="token punctuation">.</span>messages<span class="token punctuation">.</span></span><span class="token class-name">MessageListener</span><span class="token punctuation">;</span> <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span> <span class="token annotation punctuation">@NativePlugin</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">NearbyMessagesPlugin</span> <span class="token keyword">extends</span> <span class="token class-name">Plugin</span> <span class="token punctuation">{</span> <span class="token keyword">private</span> <span class="token class-name">Message</span> mMessage<span class="token punctuation">;</span> <span class="token keyword">private</span> <span class="token class-name">MessageListener</span> mMessageListener <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">MessageListener</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token annotation punctuation">@Override</span> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">onFound</span><span class="token punctuation">(</span><span class="token class-name">Message</span> message<span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token class-name">JSObject</span> result <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">JSObject</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> result<span class="token punctuation">.</span><span class="token function">put</span><span class="token punctuation">(</span><span class="token string">"message"</span><span class="token punctuation">,</span> <span class="token keyword">new</span> <span class="token class-name">String</span> <span class="token punctuation">(</span>message<span class="token punctuation">.</span><span class="token function">getContent</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">notifyListeners</span><span class="token punctuation">(</span><span class="token string">"FOUND_MESSAGE"</span><span class="token punctuation">,</span> result<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token annotation punctuation">@Override</span> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">onLost</span><span class="token punctuation">(</span><span class="token class-name">Message</span> message<span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token class-name">JSObject</span> result <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">JSObject</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> result<span class="token punctuation">.</span><span class="token function">put</span><span class="token punctuation">(</span><span class="token string">"message"</span><span class="token punctuation">,</span> <span class="token keyword">new</span> <span class="token class-name">String</span> <span class="token punctuation">(</span>message<span class="token punctuation">.</span><span class="token function">getContent</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token function">notifyListeners</span><span class="token punctuation">(</span><span class="token string">"LOST_MESSAGE"</span><span class="token punctuation">,</span> result<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token annotation punctuation">@PluginMethod</span> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token class-name">Subscribe</span><span class="token punctuation">(</span><span class="token class-name">PluginCall</span> call<span class="token punctuation">)</span><span class="token punctuation">{</span> <span class="token class-name">Nearby</span><span class="token punctuation">.</span><span class="token function">getMessagesClient</span><span class="token punctuation">(</span><span class="token function">getContext</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">subscribe</span><span class="token punctuation">(</span>mMessageListener<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token annotation punctuation">@PluginMethod</span> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token class-name">Publish</span><span class="token punctuation">(</span><span class="token class-name">PluginCall</span> call<span class="token punctuation">)</span><span class="token punctuation">{</span> <span class="token class-name">String</span> value <span class="token operator">=</span> call<span class="token punctuation">.</span><span class="token function">getString</span><span class="token punctuation">(</span><span class="token string">"message"</span><span class="token punctuation">,</span> <span class="token string">"-"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> mMessage <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Message</span><span class="token punctuation">(</span>value<span class="token punctuation">.</span><span class="token function">getBytes</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token class-name">Nearby</span><span class="token punctuation">.</span><span class="token function">getMessagesClient</span><span class="token punctuation">(</span><span class="token function">getContext</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">publish</span><span class="token punctuation">(</span>mMessage<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span></code></pre></div> <p>Event listeners are appropriate here because although we don't know when/if the phone will pick up a Bluetooth signal, a side-effect can still be triggered in the business logic when it does. <code>Subscribe</code> can be invoked to subscribe to <code>mMessageListener</code> and listen for Bluetooth messages (or we could just call <code>Subscribe</code> in the <code>onCreate</code> of our native code if we wanted to).</p> <h2>Summary</h2> <p>Once you've got a simple implementation, you can move on and write some more complex native code.</p> <p>If you want a quick-start, you can fork my <a href="https://github.com/JamesDHW/IonicNativePlugin">repository on GitHub</a>, but following along with the article and doing it yourself will really help your understanding.</p><![CDATA[RisXSS, the missing ESLint rule for React and Vue]]>/2021/07/risxss-missing-eslint-rule-react-vue//2021/07/risxss-missing-eslint-rule-react-vue/Mon, 05 Jul 2021 00:00:00 GMT<p><a href="https://en.wikipedia.org/wiki/Cross-site_scripting">XSS attacks</a> are expected to be ranked 3rd in the OWASP Top 10, compared to their 7th rank in 2021 (<a href="https://lab.wallarm.com/owasp-top-10-2021-proposal-based-on-a-statistical-data/">source</a>). The pace of application creations is increasing and so is the surface of attacks, so making your applications secure is more and more challenging and needed. React and Vue are some of the most used frameworks to build applications and include built-in protection for some types of XSS by escaping user input by default. However, there may be some cases when they need to display user content as raw HTML (a blog post with style for instance).</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">BlogPost</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> post <span class="token punctuation">}</span></span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token punctuation">(</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>div</span> <span class="token attr-name">dangerouslySetInnerHTML</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>post<span class="token punctuation">}</span></span> <span class="token punctuation">/></span></span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre></div> <p>(All the examples in this article use React but are similar with Vue).</p> <h2>A first attempt to prevent XSS attacks</h2> <p>This opens a large hole in the XSS protection. A secure solution is to first sanitize the content with a library that will remove unsafe parts of the content while keeping useful tags and attributes (for styling purpose for instance). The most popular libraries for this are <a href="https://github.com/cure53/DOMPurify">DOMPurify</a> and <a href="https://github.com/leizongmin/js-xss">js-xss</a>.</p> <p>This solution is not always known though. On a website I worked on, I saw the use of <code>dangerouslySetInnerHTML</code> that was not properly sanitized and introduced a real XSS vulnerability. My first reaction back then was: "How can I make sure this will never happen again?" So I searched the Internet and I happily found ESLint rules designed precisely to prevent the use of <a href="https://github.com/yannickcr/eslint-plugin-react/blob/master/docs/rules/no-danger.md"><code>dangerouslySetInnerHTML</code></a> and <a href="https://eslint.vuejs.org/rules/no-v-html.html"><code>v-html</code></a>. I added these rules on my projects and some time later, again, I found another XSS vulnerability.</p> <p>How did it happen? Easy: some developer wanted to display sanitized content with style, so they used <code>v-html</code> while sanitizing the output first and added a comment to disable the ESLint rule for this line. This is correct in this case. Then, another developer copied-pasted (who does not?) this code in another page, but the content was not sanitized.</p> <blockquote> <p><strong>When you have no other means than to disable a rule to do something, disabling the rule becomes the norm, and the rule loses its purpose.</strong></p> </blockquote> <h2>Custom ESLint rule to the rescue</h2> <p>This is when I talked with some colleagues and started a custom ESLint rule to warn about insecure HTML only if the HTML is not sanitized. Corentin Normand and Guillaume Klaus implemented the logic (and even wrote a <a href="https://blog.theodo.com/2020/04/create-your-own-eslint-rules/">tutorial</a> about it) and it became RisXSS.</p> <p>RisXSS warns you about insecure use of <code>dangerouslySetInnerHTML</code> and <code>v-html</code> but only if the content is not sanitized. The following snippet will not display a warning:</p> <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx"><span class="token keyword">import</span> <span class="token punctuation">{</span> sanitize <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'dompurify'</span><span class="token punctuation">;</span> <span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">BlogPost</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> post <span class="token punctuation">}</span></span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token punctuation">(</span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>div</span> <span class="token attr-name">dangerouslySetInnerHTML</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span><span class="token punctuation">{</span> __html<span class="token operator">:</span> <span class="token function">sanitize</span><span class="token punctuation">(</span>post<span class="token punctuation">)</span> <span class="token punctuation">}</span><span class="token punctuation">}</span></span> <span class="token punctuation">/></span></span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre></div> <p>By offering a smart ESLint rule, RisXSS achieves the best of both worlds:</p> <ul> <li>preventing XSS</li> <li>allowing developers to display content with style</li> </ul> <p>Now that you understand the need of RisXSS, go check out how to install it on your project on the <a href="https://github.com/theodo/RisXSS">RisXSS homepage</a>.</p><![CDATA[How to Choose Between Different Code Reusable Pattern in Vue.js?]]>/2021/06/choose-your-vue-code-reuse-pattern//2021/06/choose-your-vue-code-reuse-pattern/Mon, 28 Jun 2021 00:00:00 GMT<p>As developers at <a href="https://www.theodo.fr">Theodo</a>, we are always looking for efficient ways to reuse our code without repeating ourselves (DRY principle). In the context of an e-commerce project using the <a href="https://www.vuestorefront.io/">Vue Storefront</a> framework and Vue 2, we had the opportunity to study the best ways to reuse functional behaviors in Vue.</p> <p>We found a lot of solutions in the Vue documentation such as Mixins, composition API and scoped slots. I hope this article will help you choose yours according to your projects.</p> <p>Here are the key questions I will answer thereafter:</p> <ul> <li>What is the best method to share logic among code in Vue 2?</li> <li>What are the advantages and drawbacks of using the newly developed composition API, scoped slots, or Mixins to reuse some logic?</li> <li>When should you use one over the other?</li> </ul> <h2>TL;DR:</h2> <ul> <li>Using Mixins is one of the ways to manage shareable logic in Vue. But they have many drawbacks and should be avoided in both Vue 2 and Vue 3.</li> <li>Scoped slots are a great alternative to Mixins for Vue 2 and they still have their place in Vue 3.</li> <li>The Vue 3 composition API is a powerful tool to create readable component shareable logic. It is the option I would recommend to use in most cases but is for now limited to Vue 3.</li> </ul> <h2>Context</h2> <p>We had the responsibility to develop different authentication connections for a web platform (Google &#x26; Facebook OAuth in addition to a basic email/password authentication).</p> <p>In short, we needed to create three buttons that trigger three different actions. Our buttons looked very similar so we wanted to avoid duplicating the UI code. We thus created a single <code>LoginButton.vue</code> taking 2 props: <code>label</code> and <code>icon</code>. The button triggers an event when clicked (event named <code>click</code>).</p> <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html">// components/LoginButton.vue <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>template</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>button</span> <span class="token attr-name">@click</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>$emit(<span class="token punctuation">'</span>click<span class="token punctuation">'</span>)<span class="token punctuation">"</span></span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>img</span> <span class="token attr-name">v-if</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>icon<span class="token punctuation">"</span></span> <span class="token attr-name">:src</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>icon.src<span class="token punctuation">"</span></span> <span class="token attr-name">alt</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span><span class="token punctuation">"</span></span><span class="token punctuation">/></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>span</span><span class="token punctuation">></span></span>{{ label }}<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>span</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>button</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>template</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>script</span><span class="token punctuation">></span></span><span class="token script"><span class="token language-javascript"> <span class="token keyword">export</span> <span class="token keyword">default</span> <span class="token punctuation">{</span> props<span class="token operator">:</span> <span class="token punctuation">{</span> icon<span class="token operator">:</span> <span class="token punctuation">{</span> type<span class="token operator">:</span> Object<span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> label<span class="token operator">:</span> <span class="token punctuation">{</span> type<span class="token operator">:</span> String<span class="token punctuation">,</span> required<span class="token operator">:</span> <span class="token boolean">true</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span> </span></span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>script</span><span class="token punctuation">></span></span></code></pre></div> <p>Our objective was to:</p> <ul> <li>build the three authentication processes using this generic login button.</li> <li>organize our three different login logics in a way that would allow them to be used anywhere in the app without repeating ourselves.</li> <li>keep the code as readable as possible</li> </ul> <p>Before even starting, we wondered if creating <strong>basic js functions</strong> for each connection methods in a separate JS file would be enough. The issue is that the implementation of an oauth connection requires access to the component lifecycle hooks (the initialization of the oauth client has to be triggered in the mounted hook). This access to the component options thus became a new requirement in our quest for the best solution.</p> <h3>First lead: Define the different connection methods within a wrapping component</h3> <p>This was our first idea. We created a login page component (<code>LoginPage.vue</code>) which uses our login button and defines three methods within it: <code>googleLogin</code>, <code>facebookLogin</code>, and <code>emailLogin</code>. These methods are called when a button triggers its <code>click</code> event.</p> <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html"><span class="token comment">&lt;!-- LoginPage.vue --></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>template</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>div</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>login-page<span class="token punctuation">"</span></span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>button-login</span> <span class="token attr-name">label</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>Login with Google<span class="token punctuation">"</span></span> <span class="token attr-name">:icon</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>googleIcon<span class="token punctuation">"</span></span> <span class="token attr-name">@click</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>loginWithGoogle<span class="token punctuation">"</span></span><span class="token punctuation">/></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>button-login</span> <span class="token attr-name">label</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>Login with Facebook<span class="token punctuation">"</span></span> <span class="token attr-name">:icon</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>facebookIcon<span class="token punctuation">"</span></span> <span class="token attr-name">@click</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>loginWithFacebook<span class="token punctuation">"</span></span><span class="token punctuation">/></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>button-login</span> <span class="token attr-name">label</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>Login with email<span class="token punctuation">"</span></span> <span class="token attr-name">:icon</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>emailIcon<span class="token punctuation">"</span></span> <span class="token attr-name">@click</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>loginWithEmail<span class="token punctuation">"</span></span><span class="token punctuation">/></span></span> <span class="token tag"><span class="token tag"><span class="token punctu