In the presentation Using Rust with Elixir for code reuse and performance by Niklas Begley, he shares his experience of combining Rust and Elixir in building Doctave, a docsis code solution. Niklas, a founder of Doctave and software engineer, discusses how he encountered technical problems with markdown processing consistency between Rust and Elixir and solved it by utilizing Elixir’s ETS for caching. He also talks about using Rustler, a library for embedding Rust code in Elixir projects, to improve performance and achieve code reuse. The team encountered issues with caching and memory pressure, which were addressed by using Rust for markdown rendering and integrating it into Elixir using Rustler. Niklas emphasizes the importance of understanding the strengths and weaknesses of each language and considering the possibility of using multiple languages together for better performance. He also expresses his appreciation for Rustler and encourages anyone interested in presenting or sponsoring at an upcoming Elixir meetup to reach out to the organizers.
Using Rust with Elixir for code reuse and performance: A comprehensive overview
The genesis of doctave
Doctave was born out of a vision to create a robust and efficient static site generator, utilizing Rust for its core capabilities, coupled with a paid service built on Elixir and the Phoenix framework. Niklas Begley chose Rust for its ease of distribution, high productivity, and strong performance characteristics. On the other hand, Elixir and Phoenix were selected for their exceptional web application features, backed by the reliability of the Erlang platform. The challenge, however, was ensuring that both parts of Doctave produced consistent and identical HTML outputs.
The markdown dilemma
One of the initial challenges Niklas encountered was the inconsistency in markdown processing between Rust and Elixir. While Rust’s integration of the Cmark markdown parser allowed for extensive customization, the Elixir side — relying on Earmark — struggled with standardization, leading to differences in features and output. As the project evolved, this discrepancy began to impact performance, with Elixir spending significant time parsing markdown, which led to longer response times.
To address this, Niklas implemented a caching mechanism within Elixir using ETS (Erlang Term Storage), leveraging the concurrency and fault tolerance features inherent in the Erlang ecosystem. This approach improved performance but also introduced new challenges, such as increased memory pressure.
Enter Rustler: Bridging Rust and Elixir
To tackle the performance bottlenecks and ensure code consistency, Niklas explored using Rustler — a library that allows developers to write Elixir’s Native Implemented Functions (NIFs) in Rust. Rustler offered a safe and efficient way to inject custom library code into the Elixir/Erlang runtime, making it an ideal choice for the project.
Niklas extracted the markdown rendering logic from Doctave’s Rust-based CLI into a separate Rust crate, encapsulating the parsing and rendering functions. This crate was then integrated into the Elixir project, replacing the original Elixir parser module with a Rust-based function at runtime. This integration not only enhanced speed but also simplified the codebase, resulting in a significant boost in performance with minimal overhead.
Deployment and real-world applications
Deploying the combined Rust and Elixir application posed its own set of challenges, particularly around CI/CD processes and Dockerization. Niklas’s solution involved using Docker to build and deploy a Rust package alongside the Elixir application, creating a lightweight and efficient production environment.
The journey of integrating Rust with Elixir highlighted the importance of understanding the strengths and weaknesses of each language. Niklas demonstrated that using multiple languages within a single project can lead to substantial performance gains and simplified codebases, especially in scenarios where one language’s capabilities complement another.
Looking ahead: The power of collaboration
Niklas’s experience underscores the value of tools like Rustler, which make cross-language integration more accessible and manageable. His successful use of Rust in an Elixir project serves as an inspiration for developers facing similar challenges, showcasing how strategic language pairing can unlock new possibilities in software development.
As Niklas continues to refine Doctave and explore new frontiers in code reuse and performance optimization, his insights provide a valuable roadmap for developers seeking to harness the combined power of Rust and Elixir.