datafiles

Manage the data files created by DAPPER.

Modules:

Name	Description
uplink	Tools related to running experimentes remotely

create_run_dir(save_as, mp)

Validate save_as and create dir rc.dirs.data / save_as and sub-dirs.

The data gets saved here unless save_as is False/None.

Note: multiprocessing (locally or in the cloud) requires saving/loading data.

find_latest_run(root)

Find the latest experiment (dir containing many)

load_HMM(save_as)

Load HMM from xp.com from given dir.

load_xps(save_as)

Load xps (as a list) from given dir.

overwrite_xps(xps, save_as, nDir=100)

Save xps in save_as, but safely (by first saving to tmp).

reduce_inodes(save_as, nDir=100)

Reduce the number of xp dirs.

Done by packing multiple xps into lists (xps). This reduces the number of files (inodes) on the system, which is limited.

It also deletes files "xp.var" and "out", whose main content tends to be the printed progbar. This probably leads to some reduced loading time.

FAQ: Why isn't the default for nDir simply 1? So that we can get a progressbar when loading.

save_xps(xps, save_as, nDir=100)

Save xps (list of xps) in nDir subfolders: save_as/i.

Example

Rename attr. n_iter to nIter in some saved data:

proj_name = "Stein"
dd = rc.dirs.data / proj_name
save_as = dd / "run_2020-09-22__19:36:13"

for save_as in dd.iterdir():
    save_as = dd / save_as

    xps = load_xps(save_as)
    HMM = load_HMM(save_as)

    for xp in xps:
        if hasattr(xp,"n_iter"):
            xp.nIter = xp.n_iter
            del xp.n_iter

    overwrite_xps(xps, save_as)