Why "just read the file" gets complicated
You've got a text file. A log, a CSV, a config, whatever. You want to process it line by line: count occurrences, transform each row, look for matches. In Python you write for line in open(path): and you're done. In Rust the same operation is a few lines longer, and the reason is worth understanding before you copy-paste anything.
Reading a file in Rust forces you to think about three things up front. The OS gives you bytes, but you usually want lines of text, so something has to find the line boundaries. Reading one byte at a time is unusably slow, so you want buffering. And every read can fail (permission denied, disk error, file vanished mid-read), so the type system makes you handle that.
The standard library handles all three with BufReader and the .lines() iterator. Here's the canonical version:
use std::fs::File;
use std::io::{self, BufRead, BufReader};
fn main() -> io::Result<()> {
// File::open returns Result<File, io::Error>. ? bubbles errors to main.
let file = File::open("hello.txt")?;
// BufReader wraps the file and reads in big chunks (8 KB by default).
// Without it, every line would be a separate syscall.
let reader = BufReader::new(file);
// .lines() returns an iterator yielding Result<String, io::Error> per line.
// Each iteration reads forward in the buffer, allocating a String for the line.
for line in reader.lines() {
// Handle per-line read errors. ? exits early; you could also log and continue.
let line = line?;
println!("{line}");
}
Ok(())
}
That's the whole pattern. Three pieces: open the file, wrap in BufReader, iterate with .lines().
What each piece is doing
File::open is a thin wrapper over the OS open syscall. It returns a handle that you can read from. By itself, it has no buffering and no concept of "line." If you read directly from the File, you're talking to the kernel one read at a time.
BufReader::new(file) is the buffer. It allocates an 8 KB buffer on the heap and fills it with bytes from the file. When your code asks for data, it gets it from the buffer; the file is only touched when the buffer needs refilling. The result is dramatically fewer syscalls. For a million-line file, you make roughly a thousand reads from the kernel instead of a million.
.lines() is an iterator that walks the buffer looking for \n characters. Each time it finds one, it yields the bytes between this \n and the previous one as a String (with the \n itself stripped). It also handles \r\n line endings (Windows-style): it strips both. And it returns each line as a Result<String, io::Error>, because reading the next line might fail.
That last point matters. The outer ? on File::open only catches errors at open time. If the disk is yanked partway through reading, that error appears on a specific iteration of the loop, hence the inner let line = line?;.
A more realistic example: counting matches
Reading a file isn't usually the goal. You're reading it to do something. Counting log lines that match a pattern is a typical case:
use std::fs::File;
use std::io::{self, BufRead, BufReader};
// Count how many lines of `path` contain the substring `needle`.
fn count_matches(path: &str, needle: &str) -> io::Result<usize> {
let file = File::open(path)?;
let reader = BufReader::new(file);
let mut hits = 0usize;
for line in reader.lines() {
// ? here propagates a read error, e.g. invalid UTF-8 or IO failure.
let line = line?;
if line.contains(needle) {
hits += 1;
}
}
Ok(hits)
}
fn main() -> io::Result<()> {
let n = count_matches("server.log", "ERROR")?;
println!("found {n} ERROR lines");
Ok(())
}
Notice how the ? operator threads the error type all the way up. If anything goes wrong inside count_matches, the caller gets an io::Error. No exceptions, no hidden control flow, just a return value the compiler forces you to handle.
When the file isn't UTF-8
.lines() allocates a String for each line. A String in Rust is guaranteed to be valid UTF-8. If your file isn't UTF-8 (a binary log, a file from a Windows-1252 system, a dump with mixed encodings), .lines() will return an io::Error of kind InvalidData.
For non-UTF-8 input, switch to read_until or work with bytes directly:
use std::fs::File;
use std::io::{self, BufRead, BufReader};
fn main() -> io::Result<()> {
let file = File::open("legacy.bin")?;
let mut reader = BufReader::new(file);
let mut buf = Vec::new();
loop {
buf.clear();
// read_until reads bytes up to and including the given delimiter.
// Returns the number of bytes read, 0 on EOF.
let n = reader.read_until(b'\n', &mut buf)?;
if n == 0 { break; }
// buf now holds the line bytes (with the trailing \n if there was one).
// You can decode it however you want; this just prints the byte count.
println!("got {} bytes", buf.len());
}
Ok(())
}
This is the pattern when you can't assume UTF-8 or when you want to avoid the per-line String allocation.
Common pitfalls
A few things that bite people the first time.
Forgetting BufReader. If you write for line in file.lines(), that compiles only if you've imported BufRead, but more importantly it works only on types that already implement BufRead. A bare File doesn't. The compiler error looks like:
error[E0599]: no method named `lines` found for struct `std::fs::File`
The fix is BufReader::new(file). Without buffering, you'd be doing a syscall per byte even if you got it to compile.
Trying to mutate line and reuse the buffer. .lines() allocates a fresh String every iteration. If you're processing gigabytes of small lines, that's a lot of allocation. The optimisation is to call read_line manually and reuse a single String:
use std::io::{self, BufRead, BufReader};
use std::fs::File;
fn main() -> io::Result<()> {
let file = File::open("big.log")?;
let mut reader = BufReader::new(file);
let mut line = String::new();
loop {
// read_line appends to `line`; we have to clear it ourselves.
line.clear();
// Returns 0 on EOF.
let n = reader.read_line(&mut line)?;
if n == 0 { break; }
// Process &line here; it includes the trailing newline.
}
Ok(())
}
You're trading a bit of API noise for one allocation total instead of one per line. Worth it for hot loops, overkill for everything else.
Ignoring per-line errors. for line in reader.lines() { let line = line.unwrap(); is fine for a script but rough for a long-running service. Logging and continuing is often better than crashing on the first malformed line.
When to use what
For text files small enough to fit in memory, std::fs::read_to_string("path.txt")? and then s.lines() is even simpler and slightly faster, because there's only one syscall and no per-line allocation. The trade is that it loads the whole file at once.
For huge files, the BufReader + .lines() pattern is the right balance. For the absolute hottest loops, drop down to read_line with a reused buffer. For non-UTF-8 input, use read_until and decode yourself.