Skip to content

About

lumberjack is a log rolling package for Go

Resources

Stars

1 star

Watchers

0 watching

Forks

Β 
Β 

Repository files navigation

Lumberjack v2 (Embedded & Flash-Optimized Fork)

Go Reference Go Report Card License: MIT

An enhanced, high-performance Go rolling logger based on natefinch/lumberjack, specifically engineered for embedded systems, telemetry units, industrial controllers, and flash storage (eMMC, SD cards, SPI-NAND/NOR).


🎯 Why This Fork Exists

Standard logging packages and upstream Lumberjack are designed for server environments with hard disks or SSDs. In embedded Linux systems and IoT devices, standard logging causes critical problems:

  1. Severe Flash-Memory Wear (Write Amplification):
    • Flash storage cannot write byte-by-byte; it writes in Pages (4 KB – 16 KB) and erases in Blocks (128 KB – 4 MB).
    • Writing small log lines (50–200 bytes) frequently triggers continuous flash page rewrites, garbage collection in the Flash Translation Layer (FTL), and heavy filesystem metadata churn (mtime, inode updates).
  2. Double I/O Overhead with Post-Rotation Compression:
    • Upstream Lumberjack writes uncompressed log files to disk, and only after rotation reads the entire file back from flash to compress it and rewrite it as .gz. On flash storage, this causes double wear and high CPU spikes.
  3. Missing Hardware fsync & Buffered Control:
    • Standard loggers lack integrated, thread-safe user-space write buffering and hardware-sync (fsync) guarantees during graceful system shutdown.

πŸš€ Key Advantages & Features

Feature Upstream (natefinch/lumberjack) This Fork (themulle/lumberjack/v2)
Go Module v2 Support Partial / gopkg.in Native github.com/themulle/lumberjack/v2
Dependencies 0 external dependencies 0 external dependencies (100% Go StdLib)
Direct Gzip Compression ❌ (Post-rotation only) βœ… CompressImmediately (Streams into *gzip.Writer)
Flash-Wear Reduction ❌ (Writes directly to disk) βœ… BufferedLogger (In-Memory RAM buffer up to 99% less I/O)
Periodic Auto-Flush ❌ None βœ… WithFlushInterval (Guaranteed bounded latency)
Emergency Sync Write ❌ None βœ… WriteSync(p) for immediate FATAL/Panic flush
Hardware fsync on Close ❌ Standard file.Close() βœ… os.File.Sync() (Hardware-level flush before close)
Zap / Logger Compatibility Partial (io.WriteCloser) βœ… Full (io.WriteCloser, Sync(), Flush())

πŸ“¦ Installation

go get github.com/themulle/lumberjack/v2

Drop-in Replacement for Existing Projects

If your project or dependencies already import gopkg.in/natefinch/lumberjack.v2 and you want to use this fork without refactoring existing imports:

// in your go.mod
replace gopkg.in/natefinch/lumberjack.v2 => github.com/themulle/lumberjack/v2 v2.4.0

πŸ’‘ Usage Examples

1. Flash-Safe Embedded Logging with BufferedLogger (Recommended)

Combines in-memory write buffering with direct gzip streaming:

package main

import (
    "log"
    "time"

    "github.com/themulle/lumberjack/v2"
)

func main() {
    // 1. Configure underlying rolling logger
    rawLogger := &lumberjack.Logger{
        Filename:            "/data/logs/telemetry.log",
        MaxSize:             10,   // megabytes before rotation
        MaxBackups:          5,    // keep up to 5 archived files
        MaxAge:              30,   // days
        CompressImmediately: true, // stream directly into gzip (70-90% space reduction)
    }

    // 2. Wrap with BufferedLogger for flash-wear reduction
    flashLogger := lumberjack.NewBufferedLogger(
        rawLogger,
        lumberjack.WithBufferSize(64*1024),          // 64 KB RAM buffer (matches flash blocks)
        lumberjack.WithFlushInterval(30*time.Second), // auto-flush every 30s
    )
    defer flashLogger.Close() // flushes remaining buffer & executes hardware fsync

    log.SetOutput(flashLogger)
    log.Println("Embedded telemetry started with flash-safe logging")
}

2. Emergency Flush for Critical Errors (WriteSync)

For FATAL, PANIC, or critical event logs where data must hit persistent storage immediately:

// Writes and immediately flushes through user-space buffers to the underlying target
flashLogger.WriteSync([]byte("CRITICAL: Watchdog threshold exceeded, initiating safe stop\n"))

3. Integration with Uber Zap Logger

BufferedLogger and Logger implement Sync() error, making them directly compatible with uber-go/zap:

package main

import (
    "time"

    "github.com/themulle/lumberjack/v2"
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
)

func NewZapLogger() (*zap.Logger, func(), error) {
    rawLogger := &lumberjack.Logger{
        Filename:            "/var/log/app.log",
        MaxSize:             5,
        MaxBackups:          3,
        CompressImmediately: true,
    }

    buffered := lumberjack.NewBufferedLogger(
        rawLogger,
        lumberjack.WithBufferSize(64*1024),
        lumberjack.WithFlushInterval(15*time.Second),
    )

    writeSyncer := zapcore.AddSync(buffered)
    encoderConfig := zap.NewProductionEncoderConfig()
    core := zapcore.NewCore(
        zapcore.NewJSONEncoder(encoderConfig),
        writeSyncer,
        zap.InfoLevel,
    )

    logger := zap.New(core)
    cleanup := func() {
        _ = logger.Sync()
        _ = buffered.Close()
    }

    return logger, cleanup, nil
}

4. Graceful Shutdown & Signal Handling Guarantee

package main

import (
    "context"
    "log"
    "os"
    "os/signal"
    "syscall"

    "github.com/themulle/lumberjack/v2"
)

func main() {
    logger := lumberjack.NewBufferedLogger(&lumberjack.Logger{
        Filename:            "/data/logs/app.log",
        MaxSize:             10,
        CompressImmediately: true,
    })
    defer logger.Close() // Guaranteed fsync on exit

    log.SetOutput(logger)

    // Catch SIGINT and SIGTERM for graceful systemd / container shutdown
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    <-ctx.Done()
    log.Println("Shutting down: all buffers will be flushed and synced to flash storage")
}

βš™οΈ Configuration Reference

Logger Struct Options

Field Type Default Description
Filename string <procname>-lumberjack.log Target file path for the active log file.
MaxSize int 100 Maximum uncompressed payload size in megabytes before rotation.
MaxAge int 0 (no limit) Maximum days to retain old backup log files.
MaxBackups int 0 (all) Maximum number of rotated backup files to retain.
LocalTime bool false (UTC) If true, timestamps in backup filenames use local time.
Compress bool false Post-rotation compression (used when CompressImmediately is false).
CompressImmediately bool false New: Writes directly through *gzip.Writer into the file for 70–90% storage savings. Rotated backups automatically receive the .gz suffix (e.g. <name>-<timestamp>.log.gz) without double-suffixing or redundant re-compression.
CompressionLevel int gzip.DefaultCompression (-1) Gzip compression level (-1 = default, 1 = gzip.BestSpeed for low-power embedded CPUs, 9 = gzip.BestCompression).
CompletedDir string "" New: Finished backups (rotated and, with Compress, compressed) are moved here by rename, e.g. as an upload outbox. The active log file is never moved. Must be on the same file system. Moved files are no longer counted by MaxBackups/MaxAge.
OnCompleted func(path string) nil New: Called with the path of every finished backup (in CompletedDir after the move, if set). Runs on the mill goroutine, so it must not call Logger methods. Without CompletedDir each backup is reported once per Logger (again after a restart).

BufferedLogger Options

Functional Option Default Description
WithBufferSize(size int) 64 KB (64 * 1024) In-memory RAM buffer size. Batches small writes into flash-block aligned chunks.
WithFlushInterval(d time.Duration) 30s Auto-flush interval. Flushes in-memory data if no activity occurs within this window.
WithFlushErrorHandler(fn FlushErrorHandler) nil Optional callback invoked when an error occurs during background periodic auto-flushing.

⚑ Performance & Benchmarks

Benchmarked on Intel Core i7-1270P (16 threads), Go 1.27, Windows / Linux (77 bytes/log line):

Benchmark Scenario Latency (ns/op) Throughput (MB/s) Allocations Speedup
Standard Logger (Direct File I/O) 8,084 ns (8.1 Β΅s) 9.53 MB/s 0 B / 0 allocs Baseline
BufferedLogger (64 KB RAM Buffer) 98.2 ns 783.93 MB/s 0 B / 0 allocs ⚑ ~82Γ— faster
Standard Logger (16 Parallel Goroutines) 12,398 ns (12.4 Β΅s) 6.21 MB/s 0 B / 0 allocs Lock Contention on Disk I/O
BufferedLogger (16 Parallel Goroutines) 150.0 ns 513.46 MB/s 0 B / 0 allocs ⚑ ~83Γ— faster
CompressImmediately (Direct Gzip) 129.0 ns 597.09 MB/s 0 B / 0 allocs Streaming Gzip
BufferedLogger + CompressImmediately 127.1 ns 605.71 MB/s 0 B / 0 allocs RAM Batch + Gzip
WriteSync (Forced Hardware fsync) 285,204 ns (285 Β΅s) 0.27 MB/s 0 B / 0 allocs Hardware Persistence

Key Takeaways:

  • Zero-Allocation Hot Path: 0 B/op and 0 allocs/op across all write paths β€” completely eliminates GC pressure under high logging volume.
  • 80Γ—+ Higher Throughput: Standard loggers block all goroutines on kernel write() syscalls while holding the logger mutex. BufferedLogger only holds the lock for nanosecond in-memory memcpy operations.
  • > 99.8% Flash Write & Syscall Reduction: Batches ~870 individual log lines (75 B each) into a single 64 KB sequential write, drastically extending flash storage lifespans (eMMC, SD card).

πŸ›‘οΈ Flash Memory Protection Architecture

flowchart TD
    subgraph Application ["Application Goroutines"]
        W1[Log Write 1] --> BL[BufferedLogger]
        W2[Log Write 2] --> BL
        W3[Critical Log] -->|WriteSync| BL
    end

    subgraph RAM ["RAM (0 Flash Writes)"]
        BL --> Buf["In-Memory Buffer (bufio.Writer, e.g. 64 KB)"]
        Ticker["Auto-Flush Ticker (e.g. 30s)"] -.->|Trigger| Buf
    end

    subgraph FlashStorage ["Persistent Flash Storage"]
        Buf -->|1. Buffer Full / 2. Ticker / 3. Close| GZ["gzip.Writer (Direct Stream)"]
        GZ --> FSYNC["os.File.Sync() (Hardware fsync)"]
        FSYNC --> FILE["/data/logs/app.log"]
    end
Loading
  1. User Space (RAM): Writes are held in an in-memory buffer protected by sync.Mutex.
  2. Flash Page Alignment: When the buffer reaches 64 KB (or during auto-flush), a single sequential block is written.
  3. Zero Double-Wear: CompressImmediately streams gzip directly, avoiding the need to read and rewrite files from flash during rotation.
  4. Clean Power-Down: Calling Close() flushes user-space buffers, finalizes the gzip stream, and executes os.File.Sync() (fsync syscall) to guarantee physical persistence.

πŸ“œ License

MIT License – see the LICENSE file for details.

About

lumberjack is a log rolling package for Go

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages