An enhanced, high-performance Go rolling logger based on natefinch/lumberjack, specifically engineered for embedded systems, telemetry units, industrial controllers, and flash storage (eMMC, SD cards, SPI-NAND/NOR).
Standard logging packages and upstream Lumberjack are designed for server environments with hard disks or SSDs. In embedded Linux systems and IoT devices, standard logging causes critical problems:
- Severe Flash-Memory Wear (Write Amplification):
- Flash storage cannot write byte-by-byte; it writes in Pages (4 KB β 16 KB) and erases in Blocks (128 KB β 4 MB).
- Writing small log lines (50β200 bytes) frequently triggers continuous flash page rewrites, garbage collection in the Flash Translation Layer (FTL), and heavy filesystem metadata churn (
mtime, inode updates).
- Double I/O Overhead with Post-Rotation Compression:
- Upstream Lumberjack writes uncompressed log files to disk, and only after rotation reads the entire file back from flash to compress it and rewrite it as
.gz. On flash storage, this causes double wear and high CPU spikes.
- Upstream Lumberjack writes uncompressed log files to disk, and only after rotation reads the entire file back from flash to compress it and rewrite it as
- Missing Hardware
fsync& Buffered Control:- Standard loggers lack integrated, thread-safe user-space write buffering and hardware-sync (
fsync) guarantees during graceful system shutdown.
- Standard loggers lack integrated, thread-safe user-space write buffering and hardware-sync (
| Feature | Upstream (natefinch/lumberjack) |
This Fork (themulle/lumberjack/v2) |
|---|---|---|
| Go Module v2 Support | Partial / gopkg.in |
Native github.com/themulle/lumberjack/v2 |
| Dependencies | 0 external dependencies | 0 external dependencies (100% Go StdLib) |
| Direct Gzip Compression | β (Post-rotation only) | β
CompressImmediately (Streams into *gzip.Writer) |
| Flash-Wear Reduction | β (Writes directly to disk) | β
BufferedLogger (In-Memory RAM buffer up to 99% less I/O) |
| Periodic Auto-Flush | β None | β
WithFlushInterval (Guaranteed bounded latency) |
| Emergency Sync Write | β None | β
WriteSync(p) for immediate FATAL/Panic flush |
Hardware fsync on Close |
β Standard file.Close() |
β
os.File.Sync() (Hardware-level flush before close) |
| Zap / Logger Compatibility | Partial (io.WriteCloser) |
β
Full (io.WriteCloser, Sync(), Flush()) |
go get github.com/themulle/lumberjack/v2If your project or dependencies already import gopkg.in/natefinch/lumberjack.v2 and you want to use this fork without refactoring existing imports:
// in your go.mod
replace gopkg.in/natefinch/lumberjack.v2 => github.com/themulle/lumberjack/v2 v2.4.0Combines in-memory write buffering with direct gzip streaming:
package main
import (
"log"
"time"
"github.com/themulle/lumberjack/v2"
)
func main() {
// 1. Configure underlying rolling logger
rawLogger := &lumberjack.Logger{
Filename: "/data/logs/telemetry.log",
MaxSize: 10, // megabytes before rotation
MaxBackups: 5, // keep up to 5 archived files
MaxAge: 30, // days
CompressImmediately: true, // stream directly into gzip (70-90% space reduction)
}
// 2. Wrap with BufferedLogger for flash-wear reduction
flashLogger := lumberjack.NewBufferedLogger(
rawLogger,
lumberjack.WithBufferSize(64*1024), // 64 KB RAM buffer (matches flash blocks)
lumberjack.WithFlushInterval(30*time.Second), // auto-flush every 30s
)
defer flashLogger.Close() // flushes remaining buffer & executes hardware fsync
log.SetOutput(flashLogger)
log.Println("Embedded telemetry started with flash-safe logging")
}For FATAL, PANIC, or critical event logs where data must hit persistent storage immediately:
// Writes and immediately flushes through user-space buffers to the underlying target
flashLogger.WriteSync([]byte("CRITICAL: Watchdog threshold exceeded, initiating safe stop\n"))BufferedLogger and Logger implement Sync() error, making them directly compatible with uber-go/zap:
package main
import (
"time"
"github.com/themulle/lumberjack/v2"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
)
func NewZapLogger() (*zap.Logger, func(), error) {
rawLogger := &lumberjack.Logger{
Filename: "/var/log/app.log",
MaxSize: 5,
MaxBackups: 3,
CompressImmediately: true,
}
buffered := lumberjack.NewBufferedLogger(
rawLogger,
lumberjack.WithBufferSize(64*1024),
lumberjack.WithFlushInterval(15*time.Second),
)
writeSyncer := zapcore.AddSync(buffered)
encoderConfig := zap.NewProductionEncoderConfig()
core := zapcore.NewCore(
zapcore.NewJSONEncoder(encoderConfig),
writeSyncer,
zap.InfoLevel,
)
logger := zap.New(core)
cleanup := func() {
_ = logger.Sync()
_ = buffered.Close()
}
return logger, cleanup, nil
}package main
import (
"context"
"log"
"os"
"os/signal"
"syscall"
"github.com/themulle/lumberjack/v2"
)
func main() {
logger := lumberjack.NewBufferedLogger(&lumberjack.Logger{
Filename: "/data/logs/app.log",
MaxSize: 10,
CompressImmediately: true,
})
defer logger.Close() // Guaranteed fsync on exit
log.SetOutput(logger)
// Catch SIGINT and SIGTERM for graceful systemd / container shutdown
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
<-ctx.Done()
log.Println("Shutting down: all buffers will be flushed and synced to flash storage")
}| Field | Type | Default | Description |
|---|---|---|---|
Filename |
string |
<procname>-lumberjack.log |
Target file path for the active log file. |
MaxSize |
int |
100 |
Maximum uncompressed payload size in megabytes before rotation. |
MaxAge |
int |
0 (no limit) |
Maximum days to retain old backup log files. |
MaxBackups |
int |
0 (all) |
Maximum number of rotated backup files to retain. |
LocalTime |
bool |
false (UTC) |
If true, timestamps in backup filenames use local time. |
Compress |
bool |
false |
Post-rotation compression (used when CompressImmediately is false). |
CompressImmediately |
bool |
false |
New: Writes directly through *gzip.Writer into the file for 70β90% storage savings. Rotated backups automatically receive the .gz suffix (e.g. <name>-<timestamp>.log.gz) without double-suffixing or redundant re-compression. |
CompressionLevel |
int |
gzip.DefaultCompression (-1) |
Gzip compression level (-1 = default, 1 = gzip.BestSpeed for low-power embedded CPUs, 9 = gzip.BestCompression). |
CompletedDir |
string |
"" |
New: Finished backups (rotated and, with Compress, compressed) are moved here by rename, e.g. as an upload outbox. The active log file is never moved. Must be on the same file system. Moved files are no longer counted by MaxBackups/MaxAge. |
OnCompleted |
func(path string) |
nil |
New: Called with the path of every finished backup (in CompletedDir after the move, if set). Runs on the mill goroutine, so it must not call Logger methods. Without CompletedDir each backup is reported once per Logger (again after a restart). |
| Functional Option | Default | Description |
|---|---|---|
WithBufferSize(size int) |
64 KB (64 * 1024) |
In-memory RAM buffer size. Batches small writes into flash-block aligned chunks. |
WithFlushInterval(d time.Duration) |
30s |
Auto-flush interval. Flushes in-memory data if no activity occurs within this window. |
WithFlushErrorHandler(fn FlushErrorHandler) |
nil |
Optional callback invoked when an error occurs during background periodic auto-flushing. |
Benchmarked on Intel Core i7-1270P (16 threads), Go 1.27, Windows / Linux (77 bytes/log line):
| Benchmark Scenario | Latency (ns/op) |
Throughput (MB/s) |
Allocations | Speedup |
|---|---|---|---|---|
| Standard Logger (Direct File I/O) | 8,084 ns (8.1 Β΅s) |
9.53 MB/s | 0 B / 0 allocs | Baseline |
BufferedLogger (64 KB RAM Buffer) |
98.2 ns |
783.93 MB/s | 0 B / 0 allocs | β‘ ~82Γ faster |
| Standard Logger (16 Parallel Goroutines) | 12,398 ns (12.4 Β΅s) |
6.21 MB/s | 0 B / 0 allocs | Lock Contention on Disk I/O |
BufferedLogger (16 Parallel Goroutines) |
150.0 ns |
513.46 MB/s | 0 B / 0 allocs | β‘ ~83Γ faster |
CompressImmediately (Direct Gzip) |
129.0 ns |
597.09 MB/s | 0 B / 0 allocs | Streaming Gzip |
BufferedLogger + CompressImmediately |
127.1 ns |
605.71 MB/s | 0 B / 0 allocs | RAM Batch + Gzip |
WriteSync (Forced Hardware fsync) |
285,204 ns (285 Β΅s) |
0.27 MB/s | 0 B / 0 allocs | Hardware Persistence |
- Zero-Allocation Hot Path:
0 B/opand0 allocs/opacross all write paths β completely eliminates GC pressure under high logging volume. - 80Γ+ Higher Throughput: Standard loggers block all goroutines on kernel
write()syscalls while holding the logger mutex.BufferedLoggeronly holds the lock for nanosecond in-memorymemcpyoperations. - > 99.8% Flash Write & Syscall Reduction: Batches ~870 individual log lines (75 B each) into a single 64 KB sequential write, drastically extending flash storage lifespans (eMMC, SD card).
flowchart TD
subgraph Application ["Application Goroutines"]
W1[Log Write 1] --> BL[BufferedLogger]
W2[Log Write 2] --> BL
W3[Critical Log] -->|WriteSync| BL
end
subgraph RAM ["RAM (0 Flash Writes)"]
BL --> Buf["In-Memory Buffer (bufio.Writer, e.g. 64 KB)"]
Ticker["Auto-Flush Ticker (e.g. 30s)"] -.->|Trigger| Buf
end
subgraph FlashStorage ["Persistent Flash Storage"]
Buf -->|1. Buffer Full / 2. Ticker / 3. Close| GZ["gzip.Writer (Direct Stream)"]
GZ --> FSYNC["os.File.Sync() (Hardware fsync)"]
FSYNC --> FILE["/data/logs/app.log"]
end
- User Space (RAM): Writes are held in an in-memory buffer protected by
sync.Mutex. - Flash Page Alignment: When the buffer reaches 64 KB (or during auto-flush), a single sequential block is written.
- Zero Double-Wear:
CompressImmediatelystreams gzip directly, avoiding the need to read and rewrite files from flash during rotation. - Clean Power-Down: Calling
Close()flushes user-space buffers, finalizes the gzip stream, and executesos.File.Sync()(fsyncsyscall) to guarantee physical persistence.
MIT License β see the LICENSE file for details.