When RAM runs out
You download a 12-gigabyte CSV of sensor logs. You open your editor, write a quick script to filter the rows where temperature exceeds a threshold, and hit run. In Python, the script eats all your RAM, swaps to disk, and crawls. In Rust, you instinctively call read_to_string, and the compiler doesn't complain, but your OS kills the process with an out-of-memory error. The file is bigger than your RAM. Loading it all at once isn't just slow; it's impossible.
Streaming: the conveyor belt
Processing large data in Rust relies on streaming. Instead of loading the entire dataset into memory, you treat the file like a conveyor belt. You pull one item, process it, discard it, and pull the next. The memory footprint stays constant regardless of whether the file is 1 kilobyte or 10 terabytes.
Rust makes this pattern the default through iterators and lazy evaluation. An iterator doesn't do work until you ask for the next item. You chain operations together, and the compiler fuses them into a tight loop that touches memory only once. You define the pipeline, and Rust executes it item by item.
Streaming isn't a Rust feature; it's a systems programming discipline. Rust just makes it the path of least resistance.
The minimal streaming loop
The standard library provides BufReader to wrap file handles. It reads data in chunks, reducing the number of expensive system calls. Combined with the lines() iterator, you get a clean loop that processes one line at a time.
use std::fs::File;
use std::io::{self, BufRead, BufReader};
/// Reads a file line by line without loading it all into memory.
fn process_stream() -> io::Result<()> {
// Open the file. This returns a handle, not the content.
let file = File::open("large_dataset.csv")?;
// Wrap the file in a BufReader.
// The buffer reduces system calls by reading chunks at once.
let reader = BufReader::new(file);
// lines() returns an iterator.
// It yields one line at a time, allocating only for the current line.
for line in reader.lines() {
// Unwrap the Result. If the line has invalid UTF-8, stop here.
let line = line?;
// Process the line. Memory is reused or dropped immediately.
println!("Processing: {}", line);
}
Ok(())
}
How the buffer works
When you create a BufReader, Rust allocates a small buffer on the heap, typically 8 kilobytes. The first time you request data, the reader performs a system call to fill that buffer from the disk. The lines() iterator scans the buffer for newline characters. It yields a String for each line found.
Crucially, the iterator holds a reference to the buffer, not a copy of the whole file. As you iterate, the buffer refills automatically when it runs dry. The memory usage never exceeds the buffer size plus the size of the longest line. If a line is longer than the buffer, BufReader expands the buffer dynamically to accommodate it, but this is rare for standard text files.
The buffer is your shield against disk latency. Without it, every line would trigger a system call, and your CPU would spend most of its time waiting for the storage subsystem.
Iterator fusion: zero intermediate collections
In many languages, chaining operations creates intermediate collections. A filter creates a new list, a map creates another, and so on. Rust iterators are lazy. They describe the work but don't execute it until you consume the iterator.
When you write reader.lines().filter(|l| l.contains("error")).map(|l| l.len()), no vectors are created. The compiler generates a single loop that checks the predicate and computes the length for each line in one pass. This is called iterator fusion. It eliminates allocation overhead and improves cache locality.
Chain your operations. Collect only at the end.
Realistic parsing with allocation control
Real datasets often require parsing structured fields. A naive approach might split every line into a vector of parts. That allocates a vector for every row, which adds up quickly. You can avoid this by using iterator adapters that work on slices.
use std::fs::File;
use std::io::{BufRead, BufReader};
/// Counts rows where the third column exceeds a threshold.
/// Avoids allocating vectors by using iterator adapters.
fn count_hot_rows(path: &str, threshold: f64) -> Result<usize, std::io::Error> {
let file = File::open(path)?;
let reader = BufReader::new(file);
let mut count = 0;
let mut is_header = true;
for line_result in reader.lines() {
let line = line_result?;
if is_header {
is_header = false;
continue;
}
// split() returns an iterator.
// nth(2) advances the iterator to the third element.
// This avoids allocating a Vec of all parts.
let third_part = line.split(',').nth(2);
if let Some(part) = third_part {
// parse() returns a Result.
// unwrap_or handles parse errors by skipping bad values.
let value: f64 = part.trim().parse().unwrap_or(f64::NAN);
if value.is_nan() {
continue;
}
if value > threshold {
count += 1;
}
}
}
Ok(count)
}
The Rust community convention for CSV processing is the csv crate. It streams records efficiently and handles edge cases like quoted fields containing commas. Writing a manual parser works for simple data, but csv is the production standard. Add csv = "1.3" to your Cargo.toml and use csv::Reader::from_path.
Profile before you optimize. lines() is fast enough for 99% of use cases. Reach for manual parsing only when profiling proves the overhead is unacceptable.
Zero-allocation parsing for hot paths
The lines() iterator allocates a new String for every line. For massive files with millions of rows, this allocation pressure can become a bottleneck. If you need zero allocations per line, use BufRead::read_line with a reusable buffer.
use std::fs::File;
use std::io::{BufRead, BufReader};
/// Processes lines with zero allocations per line.
/// Reuses a single buffer for all lines.
fn process_zero_alloc() -> std::io::Result<()> {
let file = File::open("large_dataset.csv")?;
let mut reader = BufReader::new(file);
// Allocate the buffer once.
let mut buffer = String::new();
loop {
// read_line appends to the buffer.
// It returns the number of bytes read.
let bytes_read = reader.read_line(&mut buffer)?;
// If bytes_read is zero, we hit EOF.
if bytes_read == 0 {
break;
}
// Process buffer as needed.
// Note: buffer retains capacity for the next line.
println!("Line: {}", buffer.trim_end());
// Clear the content, but keep the capacity.
buffer.clear();
}
Ok(())
}
The convention here is to use buffer.clear() rather than buffer.truncate(0). Both work, but clear is idiomatic and signals intent. The buffer retains its allocated capacity across iterations, so the allocator only runs once. If you forget to pass &mut buffer to read_line, the compiler rejects you with E0596 (cannot borrow as mutable).
Reuse what you can. The allocator is slow.
Pitfalls and compiler traps
The most common mistake is calling read_to_string. This loads the entire file into a String. If the file is 10 GB and you have 8 GB of RAM, the process crashes. The compiler won't stop you here because the size is a runtime property. You have to enforce the streaming pattern yourself.
Another trap is breaking the iterator chain with collect. If you write reader.lines().collect::<Vec<_>>(), you force the iterator to run immediately and store every line in memory. You just recreated the out-of-memory bug. Keep the data flowing through the pipeline.
If you pass a BufReader where a &str is expected, you get E0308 (mismatched types). Remember that BufReader<File> implements BufRead, not AsRef<str>. You must iterate or read into a buffer.
For error handling in large streams, one bad row shouldn't kill the job. Use filter_map to discard errors gracefully.
// Discard lines with invalid UTF-8 and continue processing.
for line in reader.lines().filter_map(|result| result.ok()) {
// Process valid lines only.
}
If you collect, you've already lost. Keep the iterator alive.
Decision matrix
Use BufReader when you need to stream text data and control memory usage. It provides a simple buffer that reduces system calls and keeps the footprint constant.
Use read_to_string when the file is small enough to fit comfortably in RAM and you need random access to the content. The threshold is subjective, but anything under 100 megabytes usually qualifies on modern machines.
Use the csv crate when parsing structured comma-separated data. It handles quoting, escaping, and type conversion safely. Manual parsing is fragile and slower.
Use read_line with a reusable buffer when profiling shows allocation pressure from lines(). This eliminates per-line allocations at the cost of slightly more verbose code.
Use memory mapping when processing binary files or extremely large text files on Unix-like systems. mmap lets the OS handle paging, and you access the file as a slice of memory. This is advanced and platform-dependent, but it offers the highest throughput for read-heavy workloads.
Pick the tool that matches your data size and structure. Don't overcomplicate small files, and don't crash on big ones.