Unified Memory Management

Unified Memory Management

Shared memory for execution and caching instead of exclusive division of the regions.

Memory Management in Spark: < 1.5

·        Two separate memory managers:

o   Execution memory: computation of shuffles, joins, sorts, aggregations

o   Storage memory: caching and propagating internal data sources across cluster

·        Issues with this:

o   Manual intervention to avoid unnecessary spilling

o   No pre-defined defaults for all workloads

o   Need to partition the execution (shuffle) memory and cache memory fractions

·        Goal: Unify these two memory regions and borrow from each other

Unified Memory Management in Spark 1.6

·        Can cross between execution and storage memory

o   When execution memory exceeds its own region, it can borrow as much of the storage space as is free and vice versa

o   Borrowed storage memory can be evicted at any time (though not execution memory at this time)

·        New configurations to be introduced

·        Under memory pressure

o   Evict cached data

o   Evict storage memory

o   Evict execution memory

·        Notes:

o   Dynamically allocated reserved storage region that execution memory cannot borrow from

o   Cached data evicted only if actual storage exceeds the dynamically allocated reserved storage region

·        Result:

o   Allows for better memory management for multi-tenancy and applications relying heavily on caching

o   No cap on storage memory nor on execution memory

o   Dynamic allocation of reserved storage memory will not require user configuration

// Execute reduceByKey that will cause spill (DBC cluster with 3 nodes, 90GB RAM)

val result = sc.parallelize(0 until 200000000).map { i => (i / 2, i) }.reduceByKey(math.max).collect()

result: Array[(Int, Int)] = Array((33966328,67932657), (3035008,6070017), (52605688,105211377), (74887208,149774417), (41864592,83729185), (67568488,135136977), (40664112,81328225), (88144280,176288561), (91081800,182163601), (93161072,186322145), (26738200,53476401), (22998384,45996769), (4177592,8355185), (63269776,126539553), (95411600,190823201), (18442528,36885057), (92182440,184364881), (67090736,134181473), (59179944,118359889))

Review Stage Details for reduceByKey job

·        Spark 1.6 completes faster than the same size Spark 1.5 cluster

·        Note the Spark 1.5 spills to memory and disk