The 500MB file that crashes your app
You have a JSON log file that's 500MB. You open it, read it into a String, and pass it to serde_json::from_str. Your program allocates 500MB for the string, then another 500MB for the parsed structure, and your memory usage spikes to over a gigabyte. The parser hangs for ten seconds while it builds a massive tree in RAM. The user clicks "Cancel" and leaves.
You don't need to hold the whole file in memory. You only need to process one event at a time. Streaming lets you read the file in chunks, parse values as they arrive, and drop them immediately. Your memory usage stays flat, regardless of whether the file is 5MB or 50GB.
Streaming keeps memory flat
Rust's serde_json crate supports streaming through serde_json::Deserializer::from_reader. Instead of loading the entire input, the deserializer reads bytes on demand from anything that implements std::io::Read. It maintains a small internal buffer and a state machine. As it encounters JSON tokens, it builds values incrementally and yields them to you one by one.
Think of a conveyor belt in a factory. You don't build a warehouse to store every item that ever passes through. You grab an item, process it, and let it go. The space on the belt stays constant. Streaming works the same way. The file stays on disk, the deserializer holds a tiny window of data, and you process values as they emerge.
This approach works for any Read source: files, network sockets, compressed streams, or even strings. The API is uniform. You get constant memory usage and low latency because processing starts before the entire input is available.
Minimal streaming example
Here is the core pattern. You wrap a reader in a Deserializer and loop until the stream ends.
use std::fs::File;
use serde_json::Deserializer;
fn main() -> Result<(), Box<dyn std::error::Error>> {
// File::open returns a handle. It does not read the content yet.
// The OS handles the actual I/O in chunks.
let file = File::open("data.json")?;
// Deserializer::from_reader creates a streaming parser.
// It buffers internally to avoid excessive system calls.
let mut deserializer = Deserializer::from_reader(file);
// Loop until the stream is exhausted.
// Each call to deserialize pulls one complete JSON value.
while let Some(value) = serde_json::Value::deserialize(&mut deserializer)? {
println!("Parsed: {value}");
}
Ok(())
}
Add these dependencies to your Cargo.toml:
[dependencies]
serde_json = "1"
serde = { version = "1", features = ["derive"] }
The loop calls Value::deserialize repeatedly. Each call advances the deserializer's internal state until it has assembled a complete JSON value. When the stream ends, deserialize returns Ok(None), and the loop exits. If the JSON is malformed, deserialize returns an Err, which the ? operator propagates up.
Convention aside: serde_json::from_reader exists, but it only reads one value and stops. If your file contains multiple values, from_reader will parse the first one and ignore the rest. Use Deserializer::from_reader whenever you expect a sequence of values.
How the deserializer buffers
The Deserializer does not read byte-by-byte. That would be incredibly slow due to system call overhead. Instead, it allocates a buffer (typically 8KB) and reads chunks from the underlying reader. As it consumes tokens from the buffer, it refills when necessary.
This means you get efficient I/O without manual buffering. You also get predictable memory usage. The deserializer holds the buffer plus whatever partial value it is currently building. If you stream a JSON Lines file with small objects, memory usage stays near the buffer size. If you stream a file with occasional 10MB objects, memory will spike briefly for that object, then drop back down.
Convention aside: Do not wrap the reader in a BufReader before passing it to Deserializer::from_reader. The deserializer already buffers. Adding a BufReader creates double buffering, wastes memory, and can hurt performance because the deserializer's internal buffer management is optimized for JSON parsing.
Real-world: JSON Lines and structs
In production, you rarely stream into serde_json::Value. Value is a dynamic enum that allocates for every string and number. If you process millions of events, those allocations add up. Instead, define a struct and deserialize directly into it. This avoids the intermediate Value and is significantly faster.
JSON Lines (.jsonl) is the standard format for streaming data. Each line is a valid JSON object. The deserializer handles the newlines automatically.
use serde::Deserialize;
use serde_json::Deserializer;
use std::fs::File;
#[derive(Deserialize, Debug)]
struct LogEntry {
timestamp: u64,
message: String,
level: String,
}
fn process_logs(path: &str) -> Result<(), Box<dyn std::error::Error>> {
let file = File::open(path)?;
let mut deserializer = Deserializer::from_reader(file);
// Deserialize directly into the struct.
// This skips the Value allocation and is much faster.
while let Some(entry) = LogEntry::deserialize(&mut deserializer)? {
if entry.level == "ERROR" {
println!("Error at {}: {}", entry.timestamp, entry.message);
}
}
Ok(())
}
This pattern scales well. You can filter, aggregate, or write to a database inside the loop. The struct lives only for the duration of the loop body. Once the iteration ends, the struct is dropped, and its memory is reclaimed. You can process terabytes of data this way on a machine with a few gigabytes of RAM.
Streaming also chains with other Read implementations. You can stream from compressed files without decompressing the whole thing first.
use flate2::read::GzDecoder;
use std::fs::File;
use serde_json::Deserializer;
fn stream_gz(path: &str) -> Result<(), Box<dyn std::error::Error>> {
let file = File::open(path)?;
// GzDecoder implements Read.
// It decompresses data on the fly as the deserializer requests bytes.
let decoder = GzDecoder::new(file);
let mut deserializer = Deserializer::from_reader(decoder);
while let Some(value) = serde_json::Value::deserialize(&mut deserializer)? {
// Process decompressed values...
}
Ok(())
}
Add flate2 = "1" to your dependencies for this example. The deserializer pulls bytes from the decoder, which decompresses chunks from the file. Memory usage stays low because only the current chunk is decompressed.
Pitfalls and compiler errors
Streaming introduces a few gotchas. The most common is forgetting that Deserializer consumes the reader. If you stream values and then try to read more data from the same file handle, you'll get garbage or an empty read. The deserializer advances the file position as it parses. Once a value is deserialized, those bytes are gone.
If you try to deserialize into a type that doesn't implement Deserialize, the compiler rejects you with E0277 (the trait bound MyType: serde::Deserialize is not satisfied). Add #[derive(Deserialize)] to your struct. If you're using a third-party type, check if it supports Deserialize or wrap it in a newtype.
Malformed JSON returns an error from deserialize. You can choose to abort or skip bad entries. Skipping is common in log processing where one corrupt line shouldn't stop the whole pipeline.
while let Some(result) = LogEntry::deserialize(&mut deserializer)? {
match result {
Ok(entry) => process(entry),
Err(e) => eprintln!("Skipping bad entry: {e}"),
}
}
Wait, that code has a bug. deserialize returns Result<Option<T>, Error>. The ? unwraps the Result, so you get Option<T>. If you want to handle errors per-entry, you need to match on the Result before unwrapping.
loop {
match LogEntry::deserialize(&mut deserializer) {
Ok(Some(entry)) => process(entry),
Ok(None) => break, // Stream ended
Err(e) => eprintln!("Bad JSON: {e}"), // Skip and continue
}
}
This pattern lets you recover from errors. The deserializer's state might be corrupted after a bad value, so skipping works best when values are separated by whitespace or newlines. If the error occurs in the middle of a value, the deserializer might need to consume more bytes to resync. For robust streaming, ensure your data format has clear boundaries, like JSON Lines.
Another pitfall is holding references. If you deserialize a struct that contains &str lifetimes tied to the deserializer, you'll run into borrow checker issues. serde_json's streaming deserializer does not support zero-copy deserialization into borrowed strings because the buffer is reused. Every string you deserialize will be owned (String). If you need zero-copy parsing, load the data into a String or Vec<u8> and use serde_json::from_str or from_slice. Streaming trades zero-copy for memory efficiency.
Trust the stream. If your memory usage spikes, you're holding onto references you shouldn't. Drop values as soon as you're done with them.
Choosing the right parser
Pick the API that matches your data shape and memory constraints.
Use serde_json::from_reader when the file contains a single JSON document and you want the simplest API.
Use Deserializer::from_reader when the file contains a sequence of values, such as JSON Lines or concatenated objects, and you need to process them one by one.
Use serde_json::from_str or from_slice when the data fits comfortably in memory and you need random access to the parsed structure.
Use a custom std::io::Read wrapper when you need to transform the stream on the fly, such as decompressing gzip data or reading from a network socket.
Stream into structs for high-throughput pipelines. Value is convenient for debugging but allocates heavily. Direct deserialization is faster and uses less memory.