Skip to content

Lukas19990816/golang-json-streaming

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Streaming JSON API

A memory-efficient Golang API server that can process large JSON files using streaming parser, designed to run in a Docker container with only 50MB memory limit.

Features

  • Streaming JSON parser (memory-friendly)
  • Docker deployment with memory constraints
  • Real-time memory monitoring
  • Health check endpoint
  • Progress tracking

Quick Start

  1. Generate test data (500MB JSON file):

    make generate
    # OR manually:
    cd cmd/generate && go run main.go
  2. Build and run with Docker:

    make docker-run
    # OR manually:
    docker-compose up --build
  3. Test the API:

    # Parse with streaming (should work)
    curl http://localhost:8080/parse
    
    # Parse by loading all (will likely fail in Docker)
    curl http://localhost:8080/parse-all
    
    # Check health and memory stats
    curl http://localhost:8080/health
  4. Monitor memory usage:

    docker stats

API Endpoints

  • GET /parse - Parse the large JSON file using streaming parser (memory efficient)
  • GET /parse-all - Parse the large JSON file by loading all into memory (will likely fail with OOM)
  • GET /health - Health check with memory statistics

🔬 Streaming vs Non-streaming Comparison

Method Memory Usage 500MB File Docker 50MB Limit
Streaming (/parse) ~8-15MB ✅ Works ✅ Works
Load All (/parse-all) ~500MB+ ❌ OOM ❌ Container killed

Expected Output

Streaming endpoint (/parse):

{
  "method": "streaming",
  "records_parsed": 1000000,
  "duration_ms": 15234,
  "memory_stats": {
    "alloc": 8,
    "total_alloc": 127,
    "sys": 18,
    "num_gc": 45
  }
}

Non-streaming endpoint (/parse-all) in Docker:

# Likely results:
curl: (52) Empty reply from server
# OR
curl: (7) Failed to connect to localhost port 8080: Connection refused

# Docker container gets killed due to memory limit exceeded

The streaming parser stays well under 50MB while the non-streaming approach gets killed by Docker's memory limit.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Go 47.6%
  • Shell 38.3%
  • Makefile 9.5%
  • Dockerfile 4.6%