Low level interface api

blosc_init()

Initialize the Blosc library environment.

You must call this previous to any other Blosc call, unless you want Blosc to be used simultaneously in a multi-threaded environment, in which case you can use the blosc2compressctx blosc2decompressctx pair.

blosc_destroy()

Destroy the Blosc library environment.

You must call this after to you are done with all the Blosc calls, unless you have not used blosc_init() before.

blosc_compress(level, shuffle, itemsize, srcsize, src, dest, destsize)

Compress a block of data in the src buffer and returns the size of compressed block.

C signature: int blosc_compress(int clevel, int doshuffle, size_t typesize, size_t nbytes, const void *src, void *dest, size_t destsize)`

Arguments

• clevel - The desired compression level and must be a number between 0 (no compression) and 9 (maximum compression).

• doshuffle - Specifies whether the shuffle compression preconditioner should be applied or not. BLOSC_NOFILTER means not applying filters, BLOSC_SHUFFLE means applying shuffle at a byte level and BLOSC_BITSHUFFLE at a bit level (slower but may achieve better compression).

• typesize - Is the number of bytes for the atomic type in binary src` buffer. This is mainly useful for the shuffle preconditioner. For implementation reasons, only a 1 < typesize < 256 will allow the shuffle filter to work. When typesize is not in this range, shuffle will be silently disabled.

• nbytes - The number of bytes to compress in the src buffer.

• src - The buffer containing the data to compress.

• dest - The buffer where the compressed data will be put, must have at least the size of destsize.

• destsize - The size of the dest buffer. Blosc guarantees that if you set destsize to, at least, BLOSC_MAX_OVERHEAD, the compression will always succeed.

Returns

The number of bytes compressed. If src buffer cannot be compressed into destsize, the return value is zero and you should discard the contents of the dest buffer.

blosc_decompress(src, dest, destsize)

Decompress a block of compressed data in src, put the result in dest and returns the size of the decompressed block.

C signature: int blosc_decompress(const void *src, void *dest, size_t destsize)

Arguments

• src - The buffer to be decompressed.
• dest - The buffer where the decompressed data will be put.
• destsize = The size of the @p dest buffer

Returns

The number of bytes decompressed. If an error occurs, e.g. the compressed data is corrupted or the output buffer is not large enough, then a negative value will be returned instead.

blosc_getitem(src, start, nitems, dest)

Get nitems (of typesize size) in src buffer starting in start. The items are returned in dest buffer, which has to have enough space for storing all items.

C signature: int blosc_getitem(const void *src, int start, int nitems, void *dest)

#Arguments

• src - The compressed buffer from data will be decompressed.
• start - The zero-indexed position of the first item (of typesize size) from where data will be retrieved.
• nitems - The number of items (of typesize size) that will be retrieved.
• dest - The buffer where the decompressed data retrieved will be put.

Returns

The number of bytes copied to @p dest or a negative value if some error happens.

blosc_get_nthreads()

Returns the current number of threads that are used for compression/decompression.

blosc_set_nthreads(nthreads)

Initialize a pool of threads for compression/decompression. If nthreads is 1, then the serial version is chosen and a possible previous existing pool is ended. If this is not called, nthreads is set to 1 internally.

Returns

The previous number of threads.

blosc_get_compressor()

Get the current compressor that is used for compression.

blosc_set_compressor(compname::AbstractString)

Select the compressor to be used.

The supported ones are “blosclz”, “lz4”, “lz4hc”, “zlib” and “ztsd”. If this function is not called, then “blosclz” will be used.

#Returns

The code for the compressor (>=0). In case the compressor is not recognized, or there is not support for it in this build, it returns a -1.

blosc_set_delta(dodelta)

Select the delta coding filter to be used.

This call should always succeed.

#Arguments

• dodelta – A value >0 will activate the delta filter. If 0, it will be de-activated

blosc_set_blocksize(blocksize)

Force the use of a specific blocksize.

If 0, an automatic blocksize will be used (the default).

Warning

The blocksize is a critical parameter with important restrictions in the allowed values, so use this with care.

blosc_free_resources()

Free possible memory temporaries and thread resources.

Use this when you are not going to use Blosc for a long while.

Returns

A 0 if succeeds, in case of problems releasing the resources, it returns a negative number.

blosc_cbuffer_sizes(cbuffer)::NamedTuple{(:nbytes, :cbytes, blocksize), Tuple{Int, Int, Int}}

Get information about a compressed buffer, namely the number of uncompressed bytes (nbytes) and compressed (cbytes).

It also returns the blocksize (which is used internally for doing the compression by blocks).

You only need to pass the first BLOSCEXTENDEDHEADER_LENGTH bytes of a compressed buffer for this call to work.

This function should always succeed.

C signature void blosc_cbuffer_sizes(const void *cbuffer, size_t *nbytes, size_t *cbytes, size_t *blocksize)

blosc_cbuffer_metainfo(cbuffer)::NamedTuple{(:typesize, :flags), Tuple{Int, Int}}

Get information about a compressed buffer, namely the type size (typesize), as well as some internal flags.

You can use the BLOSC_DOSHUFFLE, BLOSC_DOBITSHUFFLE, BLOSC_DODELTA and BLOSC_MEMCPYED symbols for extracting the interesting bits (e.g. flags & BLOSC_DOSHUFFLE says whether the buffer is byte-shuffled or not).

This function should always succeed.

C signature void blosc_cbuffer_metainfo(const void *cbuffer, size_t *typesize, int *flags)

flags bits:

• bit 0: whether the shuffle filter has been applied or not

• bit 1: whether the internal buffer is a pure memcpy or not

• bit 2: whether the bitshuffle filter has been applied or not

• bit 3: whether the delta coding filter has been applied or not

blosc_cbuffer_versions(cbuffer)::NamedTuple{(:version, :versionlz), Tuple{Int, Int}}

Get information about a compressed buffer, namely the internal Blosc format version (version) and the format for the internal Lempel-Ziv compressor used (versionlz).

This function should always succeed.

C signature void blosc_cbuffer_versions(const void *cbuffer, int *version, int *versionlz)

blosc_cbuffer_complib(cbuffer)

Get the compressor library/format used in a compressed buffer.

This function should always succeed.

blosc_compcode_to_compname(compcode)

Get the compressor name associated with the compressor code.

C signature int blosc_compcode_to_compname(int compcode, const char **compname)

blosc_compname_to_compcode(compname)

Get the compressor code associated with the compressor name.

Returns

The compressor code. If the compressor name is not recognized, or there is not support for it in this build, -1 is returned instead.

blosc_list_compressors()

Get a list of compressors supported in the current build.

blosc_get_version_string()

Get the version of Blosc in string format.

blosc_get_complib_info(compname)::NamedTuple{(:code, :complib, :version), Tuple{Int, String, String}}

Get info from compression libraries included in the current build.

Returns

NamedTuple with following fields:

• code for the compression library (>=0), if it is not supported, then -1

• complib – The string where the compression library name, if available, else empty string

• version – The string where the compression library version, if available, else empty string

C signature int blosc_get_complib_info(const char *compname, char **complib, char **version)

blosc2_cparams

The parameters for creating a context for compression purposes.

In parenthesis it is shown the default value used internally when a 0 (zero) in the fields of the struct is passed to a function.

preffiler must match blosc2prefilterfn defenition: typedef int (blosc2prefilterfn)(blosc2prefilterparams params);

blosc2_dparams

The parameters for creating a context for decompression purposes. In parenthesis it is shown the default value used internally when a 0 (zero) in the fields of the struct is passed to a function.

blosc2_context

Context opaque object

blosc2_create_cctx(cparams::blosc2_cparams = blosc2_cparams())

Create a context for *_ctx() compression functions.

Returns

A pointer to the new context. NULL is returned if this fails.

blosc2_create_dctx(dparams::blosc2_dparams = blosc2_dparams())

Create a context for *_ctx() decompression functions.

Returns

A pointer to the new context. NULL is returned if this fails.

blosc2_free_ctx(context::Ptr{blosc2_context})

Free the resources associated with a context.

blosc2_compress_ctx(ctx::Ptr{blosc2_context}, src, srcsize, dest, destsize)

Context interface to Blosc compression.

This does not require a call to blosc_init and can be called from multithreaded applications without the global lock being used, so allowing Blosc be executed simultaneously in those scenarios.

Parameters

• context – A blosc2_context struct with the different compression params.

• src – The buffer containing the data to be compressed.

• srcsize – The number of bytes to be compressed from the src buffer.

• dest – The buffer where the compressed data will be put.

• destsize – The size in bytes of the dest buffer.

Returns The number of bytes compressed. If src buffer cannot be compressed into destsize, the return value is zero and you should discard the contents of the dest buffer. A negative return value means that an internal error happened. It could happen that context is not meant for compression (which is stated in stderr). Otherwise, please report it back together with the buffer data causing this and compression settings.

blosc2_decompress_ctx(ctx::Ptr{blosc2_context}, src, srcsize, dest, destsize)

Context interface to Blosc decompression.

This does not require a call to blosc_init and can be called from multithreaded applications without the global lock being used, so allowing Blosc be executed simultaneously in those scenarios.

Returns

The number of bytes copied to dest or a negative value if some error happens.

blosc2_getitem_ctx(ctx::Ptr{blosc2_context}, src, srcsize, start, nitems, dest, destsize)

Context interface counterpart for Lib.blosc_getitem.

blosc2_set_maskout(ctx::Ptr{blosc2_context}, maskout, nblocks)

Set a maskout so as to avoid decompressing specified blocks.

Remark

The maskout is valid for contexts only meant for decompressing a chunk via blosc2_decompress_ctx. Once a call to blosc2_decompress_ctx is done, this mask is reset so that next call to blosc2_decompress_ctx will decompress the whole chunk.

Parameters

• ctx – The decompression context to update.

• maskout – The boolean mask for the blocks where decompression is to be avoided.

• nblocks – The number of blocks in maskout above.

Returns If success, a 0 values is returned. An error is signaled with a negative int.

blosc2_ctx_get_cparams(ctx::Ptr{blosc2_context})

Create a cparams associated to a context.

Returns

blosc2_cparams if succeeds. Else nothing

blosc2_ctx_get_dparams(ctx::Ptr{blosc2_context})

Create a dparams associated to a context.

Returns

blosc2_cdarams if succeeds. Else nothing

blosc2_schunk

This struct is the standard container for Blosc 2 compressed data.

This is essentially a container for Blosc 1 chunks of compressed data, and it allows to overcome the 32-bit limitation in Blosc 1. Optionally, a blosc2_frame can be attached so as to store the compressed chunks contiguously.

blosc2_schunk_new(storage::blosc2_storage)::Ptr{blosc2_storage}

Create a new super-chunk.

Remark

In case that storage.urlpath is not NULL, the data is stored on-disk. If the data file(s) exist, they are overwritten.

Parameters

• storage – The storage properties.

blosc2_schunk_free(schunk::blosc2_schunk)

Release resources from a super-chunk.

Remark

All the memory resources attached to the super-frame are freed. If the super-chunk is on-disk, the data continues there for a later re-opening.

Returns

0 if success

blosc2_schunk_append_buffer(schunk::Ptr{blosc2_schunk}, src, nbytes)

Append a src data buffer to a super-chunk.

Parameters

• schunk – The super-chunk where data will be appended.

• src – The buffer of data to compress.

• nbytes – The size of the src buffer.

Returns

The number of chunks in super-chunk. If some problem is detected, this number will be negative.

blosc2_schunk_decompress_chunk(schunk::Ptr{blosc2_schunk}, nchunk, dest, nbytes)

Decompress and return the nchunk chunk of a super-chunk.

If the chunk is uncompressed successfully, it is put in the dest

Arguments

• schunk – The super-chunk from where the chunk will be decompressed.

• nchunk – The chunk to be decompressed (0 indexed).

• dest – The buffer where the decompressed data will be put.

• nbytes – The size of the dest.

Returns

The size of the decompressed chunk or 0 if it is non-initialized. If some problem is detected, a negative code is returned instead.

blosc2_schunk_get_chunk(schunk::Ptr{blosc2_schunk}, nchunk)::Vector{UInt8}

Return a compressed chunk that is part of a super-chunk in the chunk parameter.

Arguments

• schunk – The super-chunk from where to extract a chunk.

• nchunk – The chunk to be extracted (0 indexed).

C signature int blosc2_schunk_get_chunk(blosc2_schunk *schunk, int nchunk, uint8_t **chunk, bool *needs_free)

blosc2_schunk_append_chunk(schunk::Ptr{blosc2_schunk}, chunk, copy)

Append an existing chunk o a super-chunk.

Arguments

• schunk – The super-chunk where the chunk will be appended.

• chunk – The chunk to append. An internal copy is made, so chunk can be reused or freed if desired.

• copy – Whether the chunk should be copied internally or can be used as-is.

#Returns

The number of chunks in super-chunk. If some problem is detected, this number will be negative.

blosc2_schunk_insert_chunk(schunk::Ptr{blosc2_schunk}, nchunk, chunk, copy)

Insert a chunk at a specific position in a super-chunk.

Arguments

• schunk – The super-chunk where the chunk will be appended.

• nchunk – The position where the chunk will be inserted.(zero indexed)

• chunk – The chunk to insert. If an internal copy is made, the chunk can be reused or freed if desired.

• copy – Whether the chunk should be copied internally or can be used as-is.

Returns

The number of chunks in super-chunk. If some problem is detected, this number will be negative.

blosc2_schunk_update_chunk(schunk::ptr{blosc2_schunk}, nchunk, chunk, copy)

Update a chunk at a specific position in a super-chunk.

Arguments

• schunk – The super-chunk where the chunk will be updated.

• nchunk – The position where the chunk will be updated.(zero indexed)

• chunk – The new chunk. If an internal copy is made, the chunk can be reused or freed if desired.

• copy – Whether the chunk should be copied internally or can be used as-is.

Returns

The number of chunks in super-chunk. If some problem is detected, this number will be negative.

blosc2_schunk_delete_chunk(schunk::ptr{blosc2_schunk}, nchunk)

Delete a chunk at a specific position in a super-chunk.

Parameters

• schunk – The super-chunk where the chunk will be deleted.

• nchunk – The position where the chunk will be deleted. (zero indexed)

Returns

The number of chunks in super-chunk. If some problem is detected, this number will be negative.

blosc2_schunk_reorder_offsets(schunk::Ptr{blosc2_schunk}, offsets_order::Vector{Int32})

Reorder the chunk offsets of an existing super-chunk.

Arguments

• schunk – The super-chunk whose chunk offsets are to be reordered.

• offsets_order – The new order of the chunk offsets (zero ordered)

Returns

0 if suceeds. Else a negative code is returned.

blosc2_schunk_get_cparams(schunk::Ptr{blosc2_schunk})

Return the cparams associated to a super-chunk.

blosc2_schunk_get_dparams(schunk::Ptr{blosc2_schunk})

Return the dparams associated to a super-chunk.

blosc2_meta_exists(schunk::Ptr{blosc2_schunk}, name)

Find whether the schunk has a metalayer or not.

Arguments

• schunk – The super-chunk from which the metalayer will be checked.

• name – The name of the metalayer to be checked.

Returns

If successful, return the index of the metalayer. Else, return a negative value.

blosc2_meta_add(schunk::Ptr{blosc2_schunk}, name, content, content_len)

Add content into a new metalayer.

Arguments

• schunk – The super-chunk to which the metalayer should be added.

• name – The name of the metalayer.

• content – The content of the metalayer.

• content_len – The length of the content.

Returns

If successful, the index of the new metalayer. Else, return a negative value.

blosc2_meta_update(schunk::Ptr{blosc2_schunk}, name, content, content_len)

Update the content of an existing metalayer.

Arguments

• schunk – The frame containing the metalayer.

• name – The name of the metalayer to be updated.

• content – The new content of the metalayer.

• content_len – The length of the content.

Returns

If successful, the index of the metalayer. Else, return a negative value.

Note

Contrarily to blosc2_meta_add the updates to metalayers are automatically serialized into a possible attached frame.

blosc2_meta_get(schunk::Ptr{blosc2_schunk}, name)

Get the content out of a metalayer.

Arguments

• schunk – The frame containing the metalayer.

• name – The name of the metalayer to be updated.

• content – The new content of the metalayer.

• content_len – The length of the content.

Returns

If successful, the content of the metalayer. Else, return nothing.

blosc2_vmeta_exists(schunk::Ptr{blosc2_schunk}, name)

Find whether the schunk has a variable-length metalayer or not.

Arguments

• schunk – The super-chunk from which the metalayer will be checked.

• name – The name of the metalayer to be checked.

Returns

If successful, return the index of the metalayer. Else, return a negative value.

blosc2_vmeta_add(schunk::Ptr{blosc2_schunk}, name, content, content_len, cparams::blosc2_cparams)

Add content into a new variable-length metalayer.

Arguments

• schunk – The super-chunk to which the metalayer should be added.

• name – The name of the metalayer.

• content – The content of the metalayer.

• content_len – The length of the content.

• cparams – The parameters for compressing the variable-length metalayer content.

Returns

If successful, the index of the new metalayer. Else, return a negative value.

blosc2_vlmeta_update(schunk::Ptr{blosc2_schunk}, name, content, content_len)

Update the content of an existing variable-length metalayer.

Arguments

• schunk – The frame containing the metalayer.

• name – The name of the metalayer to be updated.

• content – The new content of the metalayer.

• content_len – The length of the content.

• cparams – The parameters for compressing the variable-length metalayer content.

Returns

If successful, the index of the metalayer. Else, return a negative value.

Note

Contrarily to blosc2_meta_add the updates to metalayers are automatically serialized into a possible attached frame.

blosc2_vlmeta_get(schunk::Ptr{blosc2_schunk}, name)

Get the content out of a variable-length metalayer.

Arguments

• schunk – The frame containing the metalayer.

• name – The name of the metalayer to be updated.

• content – The new content of the metalayer.

• content_len – The length of the content.

Returns

If successful, the content of the metalayer. Else, return nothing.