**Disclaimer**: This's ongoing work, so some design, interface definition, etc. might change soon as implementation continues.


From the _Scheduler_ point of view, this work is considered a _Job_ created by _API_. The job refers to doing the necessary work to enforce the new _CidConfig_. The _API_ can watch for this _Job_ state changes to see if the task of pushing a new _CidConfig_ is queued, in progress, executing, executed successfully, or failed. The _Scheduler_ also provides a human-friendly log stream of work being done for a _Cid_.

Apart from executing _API_ triggered work, like pushing a new _CidConfig_, the _Scheduler_ also does some background jobs related to deals-renewal for Cids, which have this feature enabled in their _CidConfig_. Similarly, it has background jobs for repair actions.

Every _API_ instance needs a dedicated Filecoin address that will be used to pay for actions done on the network. _Manager_ delegates wallet related activities to _WalletManager_, such as: creating new addresses for new _API_ instances, sending funds to those addresses, getting the balance.

- A Filecoin address.

It has APIs to create/update _CidConfigs_, get its address information such as balance, watch for _Job_ state changes or human-friendly Log outputs about work done by the _Scheduler_. Refer to the _CidConfig_ section to understand more about this important structure.

The main goal of this component is to do whatever its possible to reach a desired storing state for a _Cid_.

Its interface for _API_ is defined by the interface:

type Scheduler interface {

// PushConfig push a new or modified configuration for a Cid. It returns

// the JobID which tracks the current state of execution of that task.

PushConfig(APIID, string, CidConfig) (JobID, error)

// PushReplace push a new or modified configuration for a Cid, replacing

// an existing one. The replaced Cid will be unstored from the Hot Storage.

// Also it will be untracked (refer to Untrack() to understand implications)

PushReplace(APIID, string, CidConfig, cid.Cid) (JobID, error)

// GetCidInfo returns the current Cid storing state. This state may be different

// from CidConfig which is the *desired* state.

GetCidInfo(cid.Cid) (CidInfo, error)

// GetCidFromHot returns an Reader with the Cid data. If the data isn't in the Hot

// Storage, it errors with ErrHotStorageDisabled.

GetCidFromHot(context.Context, cid.Cid) (io.Reader, error)

// GetJob gets the a Job.

GetJob(JobID) (Job, error)

// WatchJobs is a blocking method that sends to a channel state updates

// for all Jobs created by an Instance. The ctx should be canceled when

// to stop receiving updates.

WatchJobs(context.Context, chan<- Job, APIID) error

// WatchLogs writes new log entries from Cid related executions.

// This is a blocking operation that should be canceled by canceling the

// provided context.

WatchLogs(context.Context, chan<- LogEntry) error

//Untrack marks a Cid to be untracked for any background processes such as

// deal renewal, or repairing.

Untrack(cid.Cid) error

}

When a new/updated _CidConfig_ is pushed by an _API_, the _Scheduler_ bounds the work of enforcing that state in a _Job_.

This _Job_ has a lifecycle: Queued, Executing, Success, Canceled, or Failed.

In summary, the _Scheduler_ is concerned about enforcing a _CidConfig_ for a Cid. It does this by inspecting the current state of the Cid in both storages, deciding on which is the necessary actions to make in both layers, and using the Hot and Cold storage APIs to execute that necessary work.

The _Scheduler_ is abstracted from particular implementations of the _Hot Storage_ and _Cold Storage_.

It relies on the following interfaces:

type HotStorage interface {

// Add adds io.Reader data ephemerally (not pinned).

Add(context.Context, io.Reader) (cid.Cid, error)

// Remove removes a stored Cid.

Remove(context.Context, cid.Cid) error

// Get retrieves a stored Cid data.

Get(context.Context, cid.Cid) (io.Reader, error)

// Store stores a Cid. If the data wasn't previously Added,

// depending on the implementation it may use internal mechanisms

// for pulling the data, e.g: IPFS network

Store(context.Context, cid.Cid) (int, error)

// Replace replaces a stored Cid with a new one. It's mostly

// thought for mutating data doing this efficiently.

Replace(context.Context, cid.Cid, cid.Cid) (int, error)

// Put adds a raw block.

Put(context.Context, blocks.Block) error

// IsStore returns true if the Cid is stored, or false

// otherwise.

IsStored(context.Context, cid.Cid) (bool, error)

}

type ColdStorage interface {

// Store stores a Cid using the provided configuration and

// account address. It returns a slice of deal errors happened

// during execution.

Store(context.Context, cid.Cid, FilConfig) (FilInfo, []DealError, error)

// Fetch fetches the cid data in the underlying storage.

Fetch(context.Context, cid.Cid, car.Store, string) error

// EnsureRenewals executes renewal logic for a Cid under a particular

// configuration. It returns a slice of deal errors happened during execution.

EnsureRenewals(context.Context, cid.Cid, FilInfo, FilConfig) (FilInfo, []DealError, error)

// IsFIlDealActive returns true if the proposal Cid is active on chain;

// returns false otherwise.

IsFilDealActive(context.Context, cid.Cid) (bool, error)

}

It also relies on a _MinerSelector_ interfaces which implement a particular strategy to fetch the most desirable N miners needed for making deals in the _Cold Storage_:

type MinerSelector interface {

// GetMiners returns a specified amount of miners that satisfy

// provided filters.

GetMiners(int, MinerSelectorFilter) ([]MinerProposal, error)

}

Particular implementations of _MinerSelector_ include:

- _FixedMiners_: which always returns a particular fixed list of miner addresses.

- _ReputationSorted_: which returns the miner addresses using a reputation system built on top of miner information.

Looking at diagram in the _Overview_ section we can see some different Hot and Cold storages:

In the first dotted box, a _Scheduler_ uses an _IPFS Node_ as the _HotStorage_ using the _CoreIPFS_ adapter as the interface implementation, which uses the _http api_ client to talk with the _IPFS node_. It also uses the _ColdFil_ adapter as the _ColdStorage_ implementation, which uses the _DealModule_ to make deals with a _Lotus instance_. It uses a _ReputationSorted_ implementation of _MinerSelector_ to fetch the best miners from a miner's reputation system.

In the second dotted box, shows another possible configuration in which uses an _IPFS Cluster_ with a _HotIpfsCluster_ adapter of _HotStorage_; or even a more advanced _HotStorage_ called _HotS3IpfsCluster_ which saves _Cid_ into _IPFS Cluster_ and some _AWS S3_ instance. The _MinerSelector_ implementation for the _ColdStorage_ is _FixedMiners_ which always returns a configured fixed list of miners to make deals with.

Cid configurations are a central part of FFS mechanics. An _API_ defines the desired state of the Cid in the Hot and Cold storage. Currently, it has the following structure:

type CidConfig struct {

// Cid is the Cid of the stored data.

Cid cid.Cid

// Hot has desired storing configuration in Hot Storage.

Hot HotConfig

// Cold has desired storing configuration in the Cold Storage.

Cold ColdConfig

}

type HotConfig struct {

// Enable indicates if Cid data is stored. If true, it will consider

// further configurations to execute actions.

Enabled bool

// AllowUnfreeze indicates that if data isn't available in the Hot Storage,

// it's allowed to be feeded by Cold Storage if available.

AllowUnfreeze bool

// Ipfs contains configuration related to storing Cid data in a IPFS node.

Ipfs IpfsConfig

}

type IpfsConfig struct {

// AddTimeout is an upper bound on adding data to IPFS node from

// the network before failing.

AddTimeout int

}

type ColdConfig struct {

// Enabled indicates that data will be saved in Cold storage.

// If is switched from false->true, it will consider the other attributes

// as the desired state of the data in this Storage.

Enabled bool

// Filecoin describes the desired Filecoin configuration for a Cid in the

// Filecoin network.

Filecoin FilConfig

}

type FilConfig struct {

// RepFactor indicates the desired amount of active deals

// with different miners to store the data. While making deals

// the other attributes of FilConfig are considered for miner selection.

RepFactor int

// DealDuration indicates the duration to be used when making new deals.

DealDuration int64

// ExcludedMiners is a set of miner addresses won't be ever be selected

// when making new deals, even if they comply to other filters.

ExcludedMiners []string

// TrustedMiners is a set of miner addresses which will be forcibly used

// when making new deals. An empty/nil list disables this feature.

TrustedMiners []string

// CountryCodes indicates that new deals should select miners on specific

// countries.

CountryCodes []string

// Renew indicates deal-renewal configuration.

Renew FilRenew

// Addr is the wallet address used to store the data in filecoin

Addr string

}

type FilRenew struct {

// Enabled indicates that deal-renewal is enabled for this Cid.

Enabled bool

// Threshold indicates how many epochs before expiring should trigger

// deal renewal. e.g: 100 epoch before expiring.

Threshold int

}

Each attribute has a description of its goal.

Both the Hot and Cold configurations have an `Enable` flag to enable/disable the Cid data storage in each of them.

If a client only wants to save data in the Cold storage, it can set `HotConfig.Enabled: false` and `ColdConfig.Enabled: true`. The same applies inversely.