UrlDownloadPipeline

UrlDownloadPipeline is an abstract class designed to handle uploads using URL-based storage adapters (UrlStorageAdapter). Unlike HashDownloadPipeline, this pipeline assumes that the delta plan (which chunks to download) is calculated externally and provided to the pipeline.

This pipeline is commonly used for remote storage systems where each chunk has a dedicated URL for download.

Constructor

Parameter	Type	Description
`storage`	`Arc<dyn UrlStorageAdapter>`	Url based storage adapter that will be used (automatically created by client).
`reconstruction`	`Arc<dyn ReconstructionService>`	ReconstructionService used for local file reconstruction.
`validation`	`Arc<dyn ValidationService>`	ValidationService used for file validation after reconstruction.
`delta`	`Arc<dyn DeltaService>`	DeltaService used for generate rd-index, comparing rd-index, etc.
`config`	`Arc<RacDeltaConfig>`	Base client configuration.

Methods

Method	Returns	Description
`execute(local_dir, urls, strategy, plan, options)`	`Result<(), DownloadError>`	Performs a full download process for a directory using urls.
`save_local_index(local_dir, index)`	`Result<(), DownloadError>`	This method will save the new local index on given folder.
`load_local_index(dir)`	`Result<RDIndex, DownloadError>`	Will create a rd-index.json of a given directory.
`find_local_index(local_dir)`	`Result<Option<RDIndex>, DownloadError>`	Will find for a rd-index.json if exists on given folder.
`download_all_missing_chunks(download_urls, target, options)`	`Result<Arc<dyn ChunkSource>, DownloadError>`	This method will download first all needed chunks via urls, and save them temporary on disk or memory.
`verify_and_delete_obsolete_chunks(plan, local_dir, remote_index, chunk_source, options)`	`Result<FileVerificationResult, DownloadError>`	This method will check for reconstructed files, verifying its hash and obsolete chunks.
`update_progress(value, phase, disk_usage, speed, options)`	`()`	This method will call progress callback given in options.
`change_state(state, options)`	`()`	This method will call state callback given in options.

DownloadOptions

The DownloadOptions object allows you to customize the behavior of a download:

pub struct DownloadOptions {
    pub force: Option<bool>,
    pub chunks_save_path: Option<PathBuf>,
    pub use_existing_index: Option<bool>,
    pub file_reconstruction_concurrency: Option<usize>,
    pub in_place_reconstruction_threshold: Option<u64>,
    pub on_progress:
        Option<Arc<dyn Fn(DownloadPhase, f64, Option<f64>, Option<f64>) + Send + Sync>>,
    pub on_state_change: Option<Arc<dyn Fn(DownloadState) + Send + Sync>>,
}

Property	Type	Description
`force`	`Option<bool>`	If true, downloads everything. If false, only new and modified chunks will be downloaded.
`chunks_save_path`	`Option<PathBuf>`	Path where chunks will be saved if `DownloadAllFirstToDisk` strategy is set.
`use_existing_index`	`Option<bool>`	If true, will search first an existing rd-index in local dir.
`file_reconstruction_concurrency`	`Option<usize>`	How many files will be reconstructed concurrently.
`in_place_reconstruction_threshold`	`Option<u64>`	Minimum file size (in bytes) required to perform an in-place reconstruction instead of using a temporary file.
`on_progress`	`Option<Arc<dyn Fn(DownloadPhase, f64, Option<f64>, Option<f64>) + Send + Sync>>`	Optional callback to inform progress.
`on_state_change`	`Option<Arc<dyn Fn(DownloadState) + Send + Sync>>`	Optional callback for state changes.

pub enum DownloadState {
    Downloading,
    Reconstructing,
    Cleaning,
    Scanning,
}

pub enum DownloadPhase {
    Download,
    Reconstructing,
    Deleting,
}

DownloadError

Custom error enum for results of DownloadPipeline. (Uses thiserror)

pub enum DownloadError {
    #[error("I/O error: {0}")]
    Io(#[from] std::io::Error),

    #[error("Delta error: {0}")]
    Delta(String),

    #[error("Storage error: {0}")]
    Storage(String),

    #[error("Index error: {0}")]
    Index(String),

    #[error("Validation error: {0}")]
    Validation(String),

    #[error("Reconstruction error: {0}")]
    Reconstruction(String),

    #[error("Invalid argument: {0}")]
    InvalidArgument(String),

    #[error("Operation aborted")]
    Aborted,

    #[error("Other error: {0}")]
    Other(String),
}

UpdateStrategy

Enum for different strategies for reconstructing and downloading files.

pub enum UpdateStrategy {
    DownloadAllFirstToMemory,
    StreamFromNetwork,
    DownloadAllFirstToDisk,
}

For more info see: Download pipeline usage

Method details

`execute(local_dir, urls, strategy, plan, options)`

Performs a full download process for a directory via urls.

Parameters

Name	Type	Description
`local_dir`	`&Path`	Directory where the new update will be downloaded.
`urls.download_urls`	`HashMap<String, ChunkUrlInfo>`	The urls identified by hash to download chunks. See ChunkUrlInfo below.
`urls.index_url`	`String`	Url to download remote rd-index.json.
`strategy`	`UpdateStrategy`	Strategy used for downloading and reconstruction. See UpdateStrategy
`plan`	`Option<DeltaPlan>`	Optional DeltaPlan for reference, if none provided will try to generate one.
`options`	`Option<DownloadOptions>`	Options for the download process. See DownloadOptions above.

pub struct ChunkUrlInfo {
    pub url: String,
    pub offset: u64,
    pub size: u64,
    pub file_path: String,
}

See ChunkUrlInfo

Returns

Result<(), DownloadError>

`save_local_index(local_dir, index)`

This method will save the new local index on given folder.

Parameters

Name	Type	Description
`local_dir`	`&Path`	Directory where the new index will be saved.
`index`	`&RDIndex`	The RDIndex object to save.

Returns

Result<(), DownloadError>

`load_local_index(dir)`

This method will create a rd-index.json of a given directory, scanning files and generating hashes.

Parameters

Name	Type	Description
`dir`	`&Path`	Directory path to generate rd-index.

Returns

Result<RDIndex, DownloadError>

`find_local_index(local_dir)`

This method will find for a rd-index.json if exists on given folder.

Parameters

Name	Type	Description
`local_dir`	`&Path`	Directory path to search rd-index.

Returns

Result<Option<RDIndex>, DownloadError>

`download_all_missing_chunks(download_urls, target, options)`

This method will download first all needed chunks for process, and save them temporary on disk or memory.

Will return ChunkSource, ChunkSources will be needed to reconstruct files, this method will ONLY return memory or disk chunk sources for offline reconstruction, if you use a storage like S3, you can omit this and use directly the StorageChunkSource with reconstruction.reconstruct_all() if you prefer.

(Using StorageChunkSource will download chunks and reconstruct file at same time, concurrently)

Parameters

Name	Type	Description
`downloadUrls`	`HashMap<String, ChunkUrlInfo>`	The urls identified by hash to download chunks. See ChunkUrlInfo above.
`target`	`DownloadTarget`	Target of the resulting ChunkSource.
`options`	`Option<DownloadOptions>`	Options for the download process. See DownloadOptions above.

pub enum DownloadTarget {
    Memory,
    Disk,
}

Returns

Result<Arc<dyn ChunkSource>, DownloadError>

`verify_and_delete_obsolete_chunks(plan, local_dir, remote_index, chunk_source, options)`

This method will check for reconstructed files, verifying its hash and obsolete chunks. If obsolete chunks are still present, it will delete them, and reconstruct file again if needed.

Parameters

Name	Type	Description
`plan`	`&DeltaPlan`	the `DeltaPlan` generated by delta.compare of the two rd-index.json for the download.
`local_dir`	`&Path`	Directory path to check files.
`remote_index`	`&RDIndex`	The remote index for reference.
`chunk_source`	`Arc<dyn ChunkSource>`	`ChunkSource` to download corrupt chunks in case of invalid files. See ChunkSource
`options`	`Option<DownloadOptions>`	Options object, see DownloadOptions above for more info (this method only uses callback)

Returns

Result<FileVerificationResult, DownloadError>

pub struct FileVerificationResult {
    pub deleted_files: Vec<String>,
    pub verified_files: Vec<String>,
    pub rebuilt_files: Vec<String>,
}

`update_progress(value, phase, disk_usage, speed, options)`

Method that will be internally used to call given on_progress callback in options.

Parameters

Name	Type	Description
`value`	`f64`	Value of the progress.
`phase`	`DownloadPhase`	Current download phase (downloading, reconstructing or deleting).
`disk_usage`	`Option<f64>`	Speed in bytes/s for reconstruction speed.
`speed`	`Option<f64>`	Speed in bytes/s for downloading phase.
`options`	`Option<&DownloadOptions>`	Options that must include the `on_progress` callback.

Returns ()

`change_state(state, options)`

Method that will be internally used to call given on_state_change callback in options.

Parameters

Name	Type	Description
`state`	`DownloadState`	Current state of the progress.
`options`	`Option<&DownloadOptions>`	Options that must include the `on_state_change` callback.

Returns ()

Constructor​

Methods​

DownloadOptions​

DownloadError​

UpdateStrategy​

Method details​

execute(local_dir, urls, strategy, plan, options)​

save_local_index(local_dir, index)​

load_local_index(dir)​

find_local_index(local_dir)​

download_all_missing_chunks(download_urls, target, options)​

verify_and_delete_obsolete_chunks(plan, local_dir, remote_index, chunk_source, options)​

update_progress(value, phase, disk_usage, speed, options)​

change_state(state, options)​

Related​

Constructor

Methods

DownloadOptions

DownloadError

UpdateStrategy

Method details

`execute(local_dir, urls, strategy, plan, options)`

`save_local_index(local_dir, index)`

`load_local_index(dir)`

`find_local_index(local_dir)`

`download_all_missing_chunks(download_urls, target, options)`

`verify_and_delete_obsolete_chunks(plan, local_dir, remote_index, chunk_source, options)`

`update_progress(value, phase, disk_usage, speed, options)`

`change_state(state, options)`

Related