In this talk, we’ll explore how the diverse and dynamic capabilities of the Elixir ecosystem effectively supported the migration of thousands of Kafka topics and clients from self-hosting to cloud at Helvetia on a large corporate scale.
Over this year-long migration, we encountered various technical roadblocks due to the volume and complexity involved. However, we successfully overcame these challenges with Elixir to ensure a non-disruptive transition experience for our hundreds of applications processing billions of messages per day.
We adopted several useful Elixir language features and patterns to build the required tooling to support this process: • Improving performance with concurrent design • Multiple data transformation and validation scripts • Directed dependency graphs • Excel reporting framework • Macros • Templating with EEx
Keywords: • Confluent Kafka Cloud • Terraform Templating • GitOps • Synchronization of multiple platforms • Credentials generation and distribution