shithub: furgit

ref: 21e2f19ae2a3baa744b6bbed997f757f0af7e01f
dir: /research/dynamic_packfiles.txt/

View raw version
dynamic packfiles to append objects

gc/refcount process punches page-sized holes in them for pages fully
within the space of unwanted objects, after setting a tombstone mark

holes are recorded in an index and re-used

then, if desired, the repack process removes all the punched holes
and anything surrounding from unwanted objects that are slightly out
of the page boundary

.idx is not a bsearch because that would cause me to need to rewrite
the entire pack every time i add objects; instead use an extendible
hash table.

genreational bloom filters


research bitcask



fetch: take current pack, remove dead objects/holes, filter objects
out, record offsets and adjust ofs_deltas since they always go
backwards, write the pack back; then stream written pack to client.
two-step necessary because pack header includes object count; could
have a custom new protocol that doesn't do so.