`pynenc_mongo.util.mongo_chunk_data`¶

MongoDB utilities for storing large data exceeding BSON limits.

Compresses data with zlib and splits across multiple chunk documents when needed. Chunk keys: “{invocation_id}:{key_prefix}[:{item}]”

Module Contents¶

Keys used in storage metadata documents.

`_compress`	Compress a string using zlib (level 6 for speed/ratio balance).
`_decompress`	Decompress zlib-compressed bytes back to a string.
`_split_into_chunks`	Split bytes into ordered chunks of at most chunk_size bytes.
`_reassemble_chunks`	Reassemble ordered chunks into the original bytes.
`_exceeds_bson_threshold`	Check if a dict exceeds the size threshold for a single BSON document.
`build_chunk_key`	Build base chunk key from named parts.
`_chunk_key`	Build chunk_key for MongoDB storage.
`_store_chunks`	Compress, split, and store data as MongoDB chunk documents.
`_retrieve_chunks`	Retrieve, reassemble, and decompress chunked data from MongoDB.
`prepare_chunk_storage`	Prepare data for storage, chunking if it exceeds threshold.
`retrieve_chunk_storage`	Retrieve data from storage, decompressing chunks if needed.

pynenc_mongo.util.mongo_chunk_data._compress(data: str) → bytes[source]¶: Compress a string using zlib (level 6 for speed/ratio balance).

pynenc_mongo.util.mongo_chunk_data._decompress(data: bytes) → str[source]¶: Decompress zlib-compressed bytes back to a string.

pynenc_mongo.util.mongo_chunk_data._split_into_chunks(data: bytes, chunk_size: int) → list[bytes][source]¶: Split bytes into ordered chunks of at most chunk_size bytes.

pynenc_mongo.util.mongo_chunk_data._reassemble_chunks(chunks: list[bytes]) → bytes[source]¶: Reassemble ordered chunks into the original bytes.

pynenc_mongo.util.mongo_chunk_data._exceeds_bson_threshold(data: dict, threshold: int) → bool[source]¶: Check if a dict exceeds the size threshold for a single BSON document.

class pynenc_mongo.util.mongo_chunk_data.StorageKey[source]¶

Keys used in storage metadata documents.

Initialization

Initialize self. See help(type(self)) for accurate signature.

pynenc_mongo.util.mongo_chunk_data.build_chunk_key(**parts: str) → str[source]¶: Build base chunk key from named parts.

pynenc_mongo.util.mongo_chunk_data._chunk_key(base_key: str, item: str = '') → str[source]¶: Build chunk_key for MongoDB storage.

pynenc_mongo.util.mongo_chunk_data._store_chunks(coll: pynenc_mongo.util.mongo_client.RetryableCollection, key: str, data: str, chunk_size: int) → int[source]¶: Compress, split, and store data as MongoDB chunk documents.

pynenc_mongo.util.mongo_chunk_data._retrieve_chunks(coll: pynenc_mongo.util.mongo_client.RetryableCollection, key: str) → str[source]¶: Retrieve, reassemble, and decompress chunked data from MongoDB.

pynenc_mongo.util.mongo_chunk_data.prepare_chunk_storage(coll: pynenc_mongo.util.mongo_client.RetryableCollection, base_key: str, data: dict[str, str], threshold: int) → dict[str, Any][source]¶: Prepare data for storage, chunking if it exceeds threshold.

pynenc_mongo.util.mongo_chunk_data.retrieve_chunk_storage(coll: pynenc_mongo.util.mongo_client.RetryableCollection, base_key: str, storage_doc: dict[str, Any]) → dict[str, str][source]¶: Retrieve data from storage, decompressing chunks if needed.