CacheGen: Store Your KV Cache on Disk or S3—Load Blazingly Fast!
TL;DR: ? CacheGen lets you store KV caches on disk or AWS S3 and load them way faster than recomputing! It compresses your KV cache up to 3× smaller than quantization so that you can load your KV cache blazingly fast while keeping response quality high. Stop wasting compute — use CacheGen to fully utilize […]
