Reference

Base.Filesystem.cpFunction
cp(src::CSeis, dst[, extents=:]; batch_size=32, workers=Distributed.workers)

Copy a CloudSeis data-set to dst and where dst is either of type Container or of type Vector{Container}. The latter is used for sharding data across multiple storage accounts.

The option extents can be used to copy a sub-set of the CloudSeis data extents (e.g. 1:10). The batch_size option allows extents to be copied in batches and where the number of extents associated with each batch is set via batch_size. Within a batch, each extent is copied via an asynchronous task.

The cp method will be executed on the set of machines defined by workers. Note that the work will be sent to the workers one batch at a time.

Base.Filesystem.mvMethod
mv(src::CSeis, dst::Container; batch_size=32, workers=Distributed.workers)

move a CloudSeis data-set to dst. See cp for a description of batch_size and workers.

Base.closeMethod
close(io::CSeis)

Close a handle to a CloudSeis data-set. This may also flush contents of the CloudSeis buffer to storage.

Base.copy!Method
copy!(ioout::CSeis, hdrsout::AbstractMatrix{UInt8}, ioin::CSeis, hdrsin::AbstractMatrix{UInt8})

copy headers from hdrsin to hdrsout, and where hdrsin are headers from a frame of ioin, and where hdrsout are headers from a frame of ioout.

Example

ioin = csopen(AzContainer("mydataset-in-cs";storageaccount="mystorageaccount"))
ioout = csopen(AzContainer("mydataset-out-cs";storageaccount="mystorageaccount"))

hdrsin = readframehdrs(ioin, 1)
hdrsout = allocframehdrs(ioout)
copy!(ioout, hdrsout, ioin, hdrsin)
Base.empty!Method
empty!(io::CSeis)

Empty (i.e. delete the extents) from a CloudSeis data-set.

Base.getMethod
get(prop::TraceProperty, hdrs::Matrix, i)

Returns the vale of a trace header property for the ith column in the trace headers hdrs.

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
hdrs = readframehdrs(io, 1)
get(prop(io, "TRACE"), hdrs, 1)
close(io)
Base.getMethod
get(prop::TraceProperty, hdr::Vector)

Returns the vale of a trace header property for the trace header hdr.

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
hdrs = readframehdrs(io,1)
hdr = @view hdrs[:,1]
get(prop(io, "TRACE"), hdr)
close(io)
Base.inMethod
in(propdef::TracePropertyDef, io::CSeis)

return true if propdef exists in the CloudSeis data-set, io.

Base.lengthMethod
length(io::CSeis)

Return the total number of frames in a CloudSeis data-set.

Base.ndimsMethod
ndims(io::CSeis)

Return the number of dimensions in a CloudSeis data-set.

Base.reduceMethod
reduce(io[; optional keyword arguments...])

Consolodate data-set extents into fewer extents.

Optional keyword arguments

  • mbytes_per_extent=1000 set the size of each output extent via either the size of each extent
  • frames_per_extent=0 set the size of each output extent via the number of frames allowed in each extent.
Base.sizeMethod
size(io::CSeis[,i])

Return the size of a CloudSeis data-set. If i is specified, then return the size of dimension i.

Base.writeMethod
write(io::CSeis, trcs, rng...)

Write traces to io, and where the size and dimension of trcs must match the size and dimensions of rng....

Example 1

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container, "w"; axis_length=[100,101,102])
write(io, rand(Float32,100,101,102), :, :, :)
close(io)

Example 2

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container, "w"; axis_lengths=[100,101,102])
write(io, rand(Float32,100,50,1), :, 1:50, 1)
close(io)
CloudSeis.cscreateMethod
cscreate(containers[; optional keyword arguments...])

Create a new CloudSeis dataset without creating a corresponding handle and without opening the data-set. Please see help for csopen for the optional keyword arguments.

containers is either of type Container or Vector{<:Container}. In the former case, all extents are stored in a single container, and in the later case, extents are sharded accross multiple containers.

In "r" and "r+" mode, and when the existing data-set is sharded over multiple containers, it is only required to supply the primary container. The "description.json" object is in the "primary" container.

CloudSeis.csopenMethod
io = csopen(containers[, mode="r"]; axis_lengths::Vector{Int}[, optional keyword arguments])

returns a handle to a CloudSeis data-set, and where container::Container corresponds to a dataset where all extents are in a single container, and container::Vector{<:Container} corresponds to a dataset where extents are sharded across mulitple containers. The container(s) can be either POSIX folders (container::Folder), or cloud storage container (e.g. container::AzContainer).

mode is one of "r" - read, "w" - write, "r+" - open existing data-set for reading and writing.

If mode="w", then axis_lenghts is a required key-word argument. axis_lengths specifies the size of the container. The axis_lengths vector is of at-least length 3.

Optional keyword arguments

  • similarto::String An existing CloudSeis dataset. If set, then all other named arguments can be used to modify the data context that belongs to the existing CloudSeis dataset.
  • datatype::String Examples are CMP, SHOT, etc. If not set, then UNKNOWN is used.
  • traceformat::DataType=Float32 Not supported. We have only tested against traceformat=Float32.
  • byteorder::String
  • extents::Vector{UnitRange} List of integer ranges (frame range) for each extent. Each extent maps to an object in block storage (set only one of extents, frames_per_extent and mbytes_per_extent).
  • frames_per_extent::Int Nominal number of frames per extent (set only one of extents, frames_per_extent and mbytes_per_extent).
  • mbytes_per_extent::Int Nominal size of each extent in units of mega-bytes (set only one of extents, frames_per_extent and mbytes_per_extent).
  • geometry::Geometry An optional three point geometry can be embedded in the CloudSeis file.
  • tracepropertydefs::Vector{TracePropertyDef} An array of trace property definitions. These are in addition to the axes property definitions.
  • dataproperties::Vector{DataProperty} An array of data properties. One property per dataset rather than one property per trace (as is true for tracepropertydefs).
  • axis_propdefs::Vector{TracePropertyDef} Trace properties corresponding to the CloudSeis axes. If not set, then SAMPLE, TRACE, FRAME, VOLUME and HYPRCUBE are used.
  • axis_units::Vector{String} Units corresponding to the CloudSeis axes. e.g. SECONDS, METERS, etc. If not set, then UNKNOWN is used.
  • axis_domains::Vector{String} Domains corresponding to the CloudSeis axes. e.g. SPACE, TIME, etc. If not set, then UNKNOWN is used.
  • axis_pstarts::Vector{Float64} Physical origins for each axis. If not set, then 0.0 is used for the physical origin of each axis.
  • axis_pincs::Vector{Float64} Physical deltas for each axis. If not set, then 1.0 is used for the physical delta for each axis.
  • axis_lstarts::Vector{Int} Logical starts for each axis. If not set, then 1 is used for the logical start for each axis.
  • axis_lincs::Vector{Int} Logical increments for each axis. If not set, then 1 is used for the logical increment for each axis.
  • compressor="leftjustify" Compress the cache before writing to disk. This is particularly useful for data with variable fold. chooose from: ("none", "blosc", "leftjustify", "zfp", "cvx")
  • compressor_options=() Pass options as key-word arguments to the compression algorithm. Currently, only "zfp" and "cvx" have options[1]

Example

Azure storage

using AzStorage, CloudSeis
container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container, "w"; axis_lengths=[10,11,12], axis_pincs=[0.004,10.0,20.0])
close(io)

POSIX storage

using AzStorage, FolderStorage
container = Folder("mydataset-cs")
io = csopen(container, "w"; axis_lengths=[10,11,12], axis_pincs=[0.004,10.0,20.0])

Notes

  • When using the similarto option, one can change the number of dimensions of the data-set via axis_lengths. If one shrinks the number of dimensions,

then various data-set properties (e.g. axis_units) will be truncated. The truncation can be customized by using appropriate key-word arguments.

  • The zfp compression options are tol, precision and rate. So, for example:
using AzStorage, CloudSeis
container = AzContainer("mydataset-cs"; storageaccount="mystorageacccount")
io = csopen(container, "w"; axis_lengths=[10,11,12], compressor="zfp", compressor_options=(tol=1e-4,))

Please refer to ZFPCompressor.jl for more information. If compressor_options is not supplied, then the defaults are (precision=16,). Also, note that compressor_options=() results in ZFP lossless compression. ZFP lossless compression will be used for the headers and foldmap regardless of the choice of compressor_options.

  • The cvx compression options are b1, b2, b3 and scale. So, for example:

Please refer to CvxCompress.jl for more information. If compressor_options is no supplied, then the defaults are (b1=16,b2=16,b3=16,scale=1e-2).

CloudSeis.datapropertyMethod
dataproperty(io::CSeis, name[, default_value])

Return the CloudSeis data property with name that is in the CloudSeis data-set io. If default_value is provided and a property with name does not exist, then return default_value.

CloudSeis.description!Method
description!(io::CSeis, kwargs...)

Limited mutation of the properties of an existing CloudSeis dataset.

Optional key-word arguments

  • axis_lengths = nothing Grow the size of the axis lengths (3rd dimension and higher)[1].

Notes

  • [1] axis_lengths must be the same length as size(io). In addition, prod(axis_lengths[3:end])

must be greater than prod(size(io)[3:end]) and axis_lengths[i] must equal size(io,i) for i∈(1..ndims(io)-1).

CloudSeis.hasdatapropertyMethod
hasdataproperty(io::CSeis, name)

Return true if the data property called name exists in io.

CloudSeis.headerlengthMethod
headerlength(io::CSeis)

Returns the length (number of bytes) of a trace header for a CloudSeis data-set.

TeaSeis.allocframeMethod
trcs,hdrs = allocframe(io::CSeis)

Allocate and return memory to store traces and trace headers for a single frame.

TeaSeis.allocframehdrsMethod
hdrs = allocframehdrs(io::CSeis)

Allocate and return memory to store trace headers for a single frame.

TeaSeis.allocframetrcsMethod
trcs = allocframetrcs(io::CSeis)

Allocate and return memory to store traces for a single frame.

TeaSeis.domainsMethod
domains(io::CSeis[,i])

Return the domain for the data-context axes. If i is specified, then return the domain for the ith data-context axis.

TeaSeis.foldMethod
fold(io::CSeis, i)

Return the fold of a CloudSeis frame corresponding, and where the frame is described by its index i. The frame index can be either an Int, a VarArg{Int}, or a CartesianIndex.

Example

io = csopen("foo.cs", "w"; axis_lengths=[10,11,12])
fold(io, 5, 6) # fold corresponding to 5th frame and 6th volume
TeaSeis.geometryMethod
geometry(io::CSeis)

Return the geometry (if any) associated with the CloudSeis data-set.

TeaSeis.leftjustify!Method
leftjustify!(io::CSeis, trcs, hdrs)

This is primarily used for compression, but one can also use this as a convenience method to move all live traces to the beginning of its frame.

TeaSeis.leftjustify!Method
leftjustify!(io::CSeis, hdrs)

Convenience method to move all live traces to the beginning of its frame.

TeaSeis.lincsMethod
lincs(io::CSeis[,i])

Return the logical increment for the data-context axes. If i is specfied, then return the logical increment for the ith data-context axis

TeaSeis.lstartsMethod
lstarts(io::CSeis[,i])

Return the logical start for the data-context axes. If i is specfied, then return the logical start for the ith data-context axis

TeaSeis.pincsMethod
pincs(io::CSeis[,i])

Return the physical increment for the data-context axes. If i is specified, then return the physical increment for the ith data-context axis.

TeaSeis.propMethod
prop(io::CSeis, propertydef)

Given a property definition propertydef, return a trace property corresponding to the CloudSeis data-set. propertydef can either be a string or a TracePropertyDef.

TeaSeis.propdefsMethod
propdefs(io::CSeis[,i])

Return the property definitions corresponding to the data-context axes. If i is specified, then return the property definition for the ith data-context axis.

TeaSeis.propsMethod
props(io::CSeis[,i])

Return the trace property corresponding to the data-context axes. If i is specified, then return the property for the ith data-context axis.

TeaSeis.pstartsMethod
pstarts(io::CSeis[,i])

Return the physical start for the data-context axes. If i is specified, then return the physical start for the ith data-context axis.

TeaSeis.readframe!Method
readframe!(io::CSeis, trcs, hdrs, idx...; regularize=true)

Read traces and headers from io into trcs::Matrix, and hdrs::Matrix for the frame idx.... idx... can either be integer(s) or a CartesianIndex.

The regularize named argument is only applicable when the compression method is LeftJustifyCompressor. If set to true, then traces are regularized to their correct context locations. Otherwise, they remain left justified. Note that one can subsequently use the regularize! method.

TeaSeis.readframeMethod
trcs,hdrs = readframe(io::CSeis, idx...; regularize=true)

Read traces and headers from io for the frame idx.... idx... can either be integer(s) or a CartesianIndex.

The regularize named argument is only applicable when the compression method is LeftJustifyCompressor. If set to true, then traces are regularized to their correct context locations. Otherwise, they remain left justified. Note that one can subsequently use the regularize! method.

TeaSeis.readframehdrs!Method
readframehdrs!(io::CSeis, trcs, idx...; regularize=true)

Read headers from io into hdrs::Matrix for the frame idx.... idx... can either be integer(s) or a CartesianIndex.

The regularize named argument is only applicable when the compression method is LeftJustifyCompressor. If set to true, then traces are regularized to their correct context locations. Otherwise, they remain left justified. Note that one can subsequently use the regularize! method.

TeaSeis.readframehdrsMethod
hdrs = readframehdrs(io::CSeis, idx...; regularize=true)

Read headers from io for the frame idx.... idx.. can either be integer(s) or a CartesianIndex.

The regularize named argument is only applicable when the compression method is LeftJustifyCompressor. If set to true, then traces are regularized to their correct context locations. Otherwise, they remain left justified. Note that one can subsequently use the regularize! method.

TeaSeis.readframetrcs!Method
readframetrcs!(io::CSeis, trcs, idx...[; regularize=true])

Read traces from io into trcs::Matrix for the frame idx.... idx... can either be integer(s) or a CartesianIndex.

The regularize named argument is only applicable when the compression method is LeftJustifyCompressor. If set to true, then traces are regularized to their correct context locations. Otherwise, they remain left justified. Note that one can subsequently use the regularize! method.

TeaSeis.readframetrcsMethod
trcs = readframetrcs(io::CSeis, idx...; regularize=true)

Read traces from io for the frame idx.... idx.. can either be integer(s) or a CartesianIndex.

The regularize named argument is only applicable when the compression method is LeftJustifyCompressor. If set to true, then traces are regularized to their correct context locations. Otherwise, they remain left justified. Note that one can subsequently use the regularize! method.

TeaSeis.readhdrs!Method
readhdrs!(io::CSeis, hdrs, rng...)

Read trace headers from io into hdrs::AbstractArray, and where the size and dimension of hdrs must correspond to the range specification in rng.... Unlike readtrcs!, we always read the entirity of each trace header. Therefore, the first range specifier must be :.

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
size(io) # (100,101,102)
hdrs = readhdrs!(io, Array{Float32,3}(undef,headerlength(io),101,1), :, :, 1)
close(io)
TeaSeis.readhdrsMethod
readhdrs(io::CSeis, rng...)

Read trace headers from io. Unlike readtrcs!, we always read the entirity of each trace header. Therefore, the first range specifier must be :.

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
size(io) # (100,101,102)
hdrs = readhdrs(io, :, :, 1)
close(io)
TeaSeis.readtrcs!Method
readtrcs!(io::CSeis, trcs, rng...)

Read traces from io into trcs::AbstractArray, and where the size and dimension of trcs must correspond to the range specification in rng....

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
size(io) # (100,101,102)
trcs = readtrcs!(io, Array{Float32,3}(undef,50,101,1), 1:50, :, 1)
close(io)
TeaSeis.readtrcsMethod
readtrcs(io::CSeis, trcs, rng...)

Read traces from io for the specified rng....

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
size(io) # (100,101,102)
trcs = readtrcs(io, 1:50, :, 1)
close(io)
TeaSeis.regularize!Method
regularize!(io::CSeis[, fld], trcs, hdrs)

This is primarily used for compression, but one can also use this as a convenience method undo the effect of leftjustify!, moving traces from the beginning of the frame to their recorded trace location.

TeaSeis.set!Method
set!(prop::TraceProperty, hdrs::Matrix, i, value)

Set the vale of the header property for the ith column of hdrs.

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
hdrs = readframehdrs(io, 1)
set!(prop(io, "TRACE"), hdr, 1, 2)
close(io)
TeaSeis.set!Method
set!(prop::TraceProperty, hdr::Vector, value)

Set the vale of a trace header property for the hdr.

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
hdrs = readframehdrs(io, 1)
hdr = @view hdrs[:,1]
set!(prop(io, "TRACE"), hdr, 1)
close(io)
TeaSeis.unitsMethod
units(io::CSeis[,i])

Return the physical unit for the data-context axes. If i is specfied, then return the physical unit for the ith data-context axis.

TeaSeis.writeframeMethod
writeframe(io::CSeis, trcs, hdrs)

Write a frame to io. The location of the frame is determined by the axis headers set in hdrs.

TeaSeis.writeframeMethod
writeframe(io::CSeis, trcs, idx...)

Write a frame to io. The location of the frame is determined from idx... which can either be integer(s) or a CartesianIndex. A minimal set of headers are created from idx... and are also written to io.

CloudSeis.LogicalIndicesType
I = LogicalIndices(io::CSeis)

Returns a construct similar to CartesianIndices that allows conversion from linear indices to cartesian indices that are offest by the logical starts and deltas of the CloudSeis data context. In addition, LogicalIndices implements iteration for looping over all frames in a data-set. For example,

io = csopen(AzContainer("foo"; storageaccount="bar))
idx = LogicalIndices(io)[2] # get the index corresponding to the second frame in the data-set.
for idx in LogicalIndices(io)
    @show idx
    trcs, hdrs = getframe(io, idx)
end