Microservices - Solve your Redis fixtures problem
April 04, 2016Maxime Thoonsen2 min read
Redis is a fantastic tool to handle data in an amazingly simple and rapid manner. If you're not using it yet, here's a way you can manage redis fixtures in your projects.
Redis and microservices
In our current web project, we use Redis to handle hot data that can be updated using microservices.
To initialize our development environment, we use scripts that populate Redis from the Postgresql databases of our microservices. Our problem: As we have quite a large set of data, it was taking time to populate Redis. This should not be the case in production, but in a development environment, this is quite common.
The quickest and most simple solution we found is to load a backup of our Redis store. Fortunately for us, Redis automatically creates a backup file (
/var/lib/redis/dump.rdb) that is constantly updated. All we have to do to restore a backup is to replace the
/var/lib/redis/dump.rdb file with our own.
As the file is quite large, we need to compress it. This is where we were met with a significant problem that was not covered in the article: the tar command corrupts the
dump.rdb file. We simply solve it by using the bzip2 command instead.
Loading the fixtures in Vagrant and Docker
If you are using Vagrant like us for your development environment, you need to run the following script inside the vm:
cd /var/www/myapp/current cp data/backupredis.rdb.bz2 data/dump.rdb.bz2 bunzip2 data/dump.rdb.bz2 sudo service redis-server stop sudo mv data/dump.rdb /var/lib/redis/dump.rdb sudo chown redis:redis /var/lib/redis/dump.rdb sudo service redis-server start
Notice that you need to run it as root. If you're not using it yet, ...
- You can build a new image with the correct dump.rdb.
- You can put the dump.rdb in a shared folder and you replace it during the start.sh of your container.
Updating the fixtures
As the project goes on, we need new versions of the fixtures.
To update our backup file with the data from one environment we use the following script:
#!/usr/bin/env bash rm -f ../data/backupredis.rdb.bz2 scp root@IP_TARGET_ENVIRONMENT:/var/lib/redis/dump.rdb ../data/backupredis.rdb bzip2 ../data/backupredis.rdb
We then commit and push the file so that developers can quickly get the last version of the fixtures. For Docker users, if have the build a new image solution, you need to rebuild the image of your container at the end of this step.
Dealing with the backup of a Redis store was quite simple - another of the reasons why Redis is an awesome tool. If you liked this article and don't want to miss the next one, you can follow me on Twitter.
Theodo - CTO