Data can be called whatever you want,hype, cash, or trash,but it doesn’t change the fact that, in today’s world, data is precious. It might change in the future, but for now, you’ve got to adapt to it.
I’m not saying you should add machine learning models to everything or process large datasets for no reason; that doesn’t make sense. If you try to build everything around data, it’s just hype. But if you use it efficiently in your projects now, you’ll truly benefit from it.
Today’s article is for those who want to process data efficiently for their next projects. Processing large datasets quickly and efficiently is more critical than ever. Before data, we used to say time is cash. Less time spent processing data equals more cash.
And now, the next question: why Rust for this? Simply put, it’s because of Rust’s performance and safety. It’s not hard to use Rust’s concurrency model to build high-performance systems for data science and machine learning.
That’s why I’ve prepared this article for those who want to use Rust to handle massive datasets, leveraging parallel processing.
Concurrency And Parallelism in Rust
Concurrency and parallelism may seem quite similar, but there’s a huge difference.
No need to get technical to understand what they do. You can think of concurrency as the ability to manage multiple tasks at once, using a single core to handle many tasks.
On the other hand, parallelism is about actually performing multiple tasks at the same time. So, instead of using one core, you use multiple cores to handle multiple tasks.
Rust is great at both concurrency and parallelism because of its memory safety and ownership system, which ensures there are no data races.
Rust makes concurrency easy by providing simple, low-cost tools that are safe and efficient.
Concurrency Example in Rust
While not the main focus of this article, I’ll briefly explain how concurrency works in Rust before diving into parallel and distributed processing.
use std::thread;
fn main() {
let tasks = vec!["task1", "task2", "task3"];
let handles: Vec<_> = tasks.into_iter().map(|task| {
thread::spawn(move || {
println!("Running: {}", task);
})
}).collect();
for handle in handles {
handle.join().unwrap();
}
}This is about managing multiple tasks at once. The tasks are executed in overlapping time periods, but not necessarily simultaneously. On a single-core processor, tasks are still managed concurrently by switching between them, but they’re not actually being executed at the same time.
If you have multiple cores in your system, threads can run on different cores, allowing them to work simultaneously.
use tokio;
#[tokio::main]
async fn main() {
let tasks = vec!["task1", "task2", "task3"];
let handles: Vec<_> = tasks.into_iter().map(|task| {
tokio::spawn(async move {
println!("Running: {}", task);
})
}).collect();
for handle in handles {
handle.await.unwrap();
}
}Same process can be done in Tokio as well, and there are some advantages. Tokio uses asynchronous execution, allowing tasks to run concurrently without blocking threads.
This makes it more efficient for I/O-bound tasks compared to std::thread, which spawns separate threads for each task. Tokio handles many tasks on a small thread pool, leading to better scalability and lower resource usage.
So, the most important difference between Tokio and standard threads is that Tokio doesn’t block threads. Unlike standard threads, which block while waiting for tasks to complete, Tokio can continue working on other tasks during that time, improving efficiency. In contrast, with standard threads, the thread waits until the current task is done before moving on.
Parallel Data Processing with Rust
Parallel data processing involves breaking a problem into smaller tasks that can be executed simultaneously.
Rust provides excellent tools for this, with libraries like Rayon that simplify parallel programming.
use rayon::prelude::*;
fn main() {
let data = vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
let squared: Vec<i32> = data.par_iter().map(|&x| x * x).collect();
println!("{:?}", squared);
}The par_iter function allows you to iterate over the data in parallel, applying the map function to each element concurrently for faster results.
Rayon takes care of splitting the data and executing the tasks in parallel.
use rayon::prelude::*;
fn main() {
let mut data = vec![5, 2, 9, 1, 4, 6];
data.par_sort();
println!("{:?}", data);
}You can also perform parallel sorting. Sorting can take a long time, but thanks to the par_sort function, you can keep it faster.
use rayon::prelude::*;
fn main() {
let data = vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
let even_numbers: Vec<i32> = data.par_iter().filter(|&&x| x % 2 == 0).cloned().collect();
println!("{:?}", even_numbers);
}Yeah, and you can also filter with par_iter, and so on.
Not just that you can also use Rayon for your run custom tasks in parallel (for example, processing chunks of data).
use rayon::prelude::*;
use rayon::ThreadPoolBuilder;
fn main() {
let pool = ThreadPoolBuilder::new().num_threads(4).build().unwrap();
pool.install(|| {
let result = (0..10).into_par_iter()
.map(|x| x * x)
.collect::<Vec<_>>();
println!("Parallel Result: {:?}", result);
});
}In this example, there’s no need to use ThreadPoolBuilder for this specific task, but it’s included just for demonstration purposes. ThreadPoolBuilder allows you to create a custom thread pool where you can adjust the number of threads according to your needs.
The install() method is used to execute tasks within the thread pool. Inside it, you can process tasks in parallel as usual with into_par_iter().
Caution! into_par_iter() is for working with the actual data (owned values). par_iter() is for working with references to the data.
Conclusion
Parallel computing in Rust is smooth and clean, making it easy to speed up your tasks. However, it’s important to note that parallel computing is not always necessary!
For simple tasks, don’t waste CPU cores. You can achieve the same results efficiently on a single core with good optimization and a bit of asynchronous processing.
Only when you notice performance issues or overloading, that’s when it’s time to consider parallel processing. Rust’s performance is already great, so don’t overuse CPU cores for simple tasks.
The sign of existence etched onto the soul.
