Growing volume of data on Kafka inspires Engie to introduce backup & restore with Kannika Armory

Engie is the Belgian market leader in electricity, natural gas, and energy services. Like most sectors, the energy industry is rapidly transforming and becoming increasingly data-driven. Recognizing the need to manage this influx of data, Engie decided to adopt Kafka. As the importance of the data on Kafka increased, they also started to see the benefit of having Kannika Armory as a dedicated backup & restore solution.

‍

With 2.8 million customers and a 150-year history in Belgium, Engie is synonymous with energy. In recent years, the company has been leading the charge toward renewable energy. The energy sector is one of the most dynamic markets, but this rapid evolution also presents new challenges. For example, digital meters record electricity consumption every 15 minutes, and solar panels influence supply and demand, significantly increasing the volume of data Engie needs to manage.

‍

Traditional integration technologies could no longer handle this load, which led to the adoption of Kafka. “As this was new for us, we sought the help of an experienced consultancy team”, says Laurens Raven, IS Software Architect EAI at Engie. “We partnered with Cymo, who explained how Kafka could benefit us. Ultimately, we opted for Confluent Cloud as our dedicated cluster for Kafka.”

‍

Early adopters of Kannika Armory

When transitioning to Kafka, many companies overlook the necessity of a backup & restore solution. “Confluent Cloud offers three replications of your data, so we initially questioned the need for a separate backup and restore solution”, Laurens explains. “But when you think about it, there are several reasons why you want to keep a backup of your data. For example, we wanted to replay our production data in our non-production environment to investigate problems and create a golden data set for new deployments. And of course, having a backup ensures we’re covered in the event of a cluster disaster.”

“Most other products offer backup and restore as a feature. Kannika was the only solution that focuses exclusively on this, which not only made it more cost-effective but also more flexible”

Engie discovered Kannika Armory through Cymo but conducted a market comparison before making a final decision. “Most other products offer backup and restore as a feature. Kannika was the only solution that focuses exclusively on this, which not only made it more cost-effective but also more flexible”, Laurens notes. “Being early adopters allowed us to provide feedback and help shape Kannika Armory into an even better solution. For example, we suggested the addition of an interface, which was later implemented to meet the needs of our operational teams.”

‍

Another notable feature is Kannika’s capability to handle retention times. "Data is only relevant for a specific period and needs to be purged after its usefulness expires. Traditional backup systems do not take this into account, so you need to eliminate everything manually. But with Kannika Armory, we can define a retention period where the backup files are deleted after the specified period in time.”

‍

‍Seamless migration from prod to non-prod

Since Engie has started working with Kannika Armory, they have already experienced multiple benefits. “When you produce data, you do this according to a schema. However, you can also produce bad data that is not in accordance with this schema. In Kafka, data can exist for a long time, if not infinite. And by using replication clusters, you’ll just replicate that bad data. With Kannika’s backup and restore system, which is operationally decoupled, we can empty a topic and reset to a previous moment when all data was correct.”

‍

"Kannika also offers seamless transition between production and non-production environments. In other solutions, you have to make manual adjustments when restoring items from production to non-production. For instance, each schema is assigned an ID, but this number changes in the non-prod environment, so you must update it for your data to function correctly. Kannika Armory eliminates the need for manual intervention.”

‍

Why Kannika?

After exploring several products, Engie is very satisfied with Kannika Armory as backup & restore solution. “In most tools, backup & restore is just one part of the product, but Kannika focuses solely on backup. They prioritized the core features before moving on to other elements like the interface”, says Laurens. “Their team is also highly responsive. We communicate regularly via Slack, and when we report a bug, it is resolved fast. They’re also quick to implement suggestions that align with their roadmap. Since we started using Kannika Armory, the solution has seen significant optimization.”

"Since we started using Kannika Armory, the solution has seen significant optimization.”

The system’s performance is also meeting Engie’s expectations. “We had a good case where we brought all production data from our core integrations to non-production. We’re talking about multiple gigabytes of data from around 60 topics. Kannika handled this perfectly. Speed is also critical, of course. Sometimes we have topics that contain 12 to 14 million messages, and Kannika completes the job in a matter of minutes. We currently see no issues with the technology.”

‍

To conclude, what is the most important lesson Engie has learned about backing up their Kafka environment? “Never underestimate backup & restore. It is more comprehensive than it appears.  Some solutions are very promising and may have an attractive interface, but if we compare them to Kannika now, we notice that they often lack important features.”

‍