API - Files

Load benchmark dataset, save and restore model, save and load variables. TensorFlow provides .ckpt file format to save and restore the models, while we suggest to use standard python file format .npz to save models for the sake of cross-platform.

# save model as .ckpt
saver = tf.train.Saver()
save_path = saver.save(sess, "model.ckpt")
# restore model from .ckpt
saver = tf.train.Saver()
saver.restore(sess, "model.ckpt")

# save model as .npz
tl.files.save_npz(network.all_params , name='model.npz')

# restore model from .npz
load_params = tl.files.load_npz(path='', name='model.npz')
tl.files.assign_params(sess, load_params, network)

# you can assign the pre-trained parameters as follow
# 1st parameter
tl.files.assign_params(sess, [load_params[0]], network)
# the first three parameters
tl.files.assign_params(sess, load_params[:3], network)
load_mnist_dataset([shape, path]) Automatically download MNIST dataset and return the training, validation and test set with 50000, 10000 and 10000 digit images respectively.
load_cifar10_dataset([shape, path, ...]) The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class.
load_ptb_dataset([path]) Penn TreeBank (PTB) dataset is used in many LANGUAGE MODELING papers, including “Empirical Evaluation and Combination of Advanced Language Modeling Techniques”, “Recurrent Neural Network Regularization”.
load_matt_mahoney_text8_dataset([path]) Download a text file from Matt Mahoney’s website if not present, and make sure it’s the right size.
load_imdb_dataset([path, nb_words, ...]) Load IMDB dataset
load_nietzsche_dataset([path]) Load Nietzsche dataset.
load_wmt_en_fr_dataset([path]) It will download English-to-French translation data from the WMT‘15 Website (10^9-French-English corpus), and the 2013 news test from the same site as development set.
load_flickr25k_dataset([tag, path, ...]) Returns a list of images by a given tag from Flick25k dataset, it will download Flickr25k from the official website at the first time you use it.
load_flickr1M_dataset([tag, size, path, ...]) Returns a list of images by a given tag from Flickr1M dataset, it will download Flickr1M from the official website at the first time you use it.
load_cyclegan_dataset([filename, path]) Load image data from CycleGAN’s database, see this link.
save_npz([save_list, name, sess]) Input parameters and the file name, save parameters into .npz file.
load_npz([path, name]) Load the parameters of a Model saved by tl.files.save_npz().
assign_params(sess, params, network) Assign the given parameters to the TensorLayer network.
load_and_assign_npz([sess, name, network]) Load model from npz and assign to a network.
save_npz_dict([save_list, name, sess]) Input parameters and the file name, save parameters as a dictionary into .npz file.
load_and_assign_npz_dict([name, sess]) Restore the parameters saved by tl.files.save_npz_dict().
save_ckpt([sess, mode_name, save_dir, ...]) Save parameters into ckpt file.
load_ckpt([sess, mode_name, save_dir, ...]) Load parameters from ckpt file.
save_any_to_npy([save_dict, name]) Save variables to .npy file.
load_npy_to_any([path, name]) Load .npy file.
file_exists(filepath) Check whether a file exists by given file path.
folder_exists(folderpath) Check whether a folder exists by given folder path.
del_file(filepath) Delete a file by given file path.
del_folder(folderpath) Delete a folder by given folder path.
read_file(filepath) Read a file and return a string.
load_file_list([path, regx, printable]) Return a file list in a folder by given a path and regular expression.
load_folder_list([path]) Return a folder list in a folder by given a folder path.
exists_or_mkdir(path[, verbose]) Check a folder by given name, if not exist, create the folder and return False, if directory exists, return True.
maybe_download_and_extract(filename, ...[, ...]) Checks if file exists in working_directory otherwise tries to dowload the file,
natural_keys(text) Sort list of string with number in human order.
npz_to_W_pdf([path, regx]) Convert the first weight matrix of .npz file to .pdf by using tl.visualize.W().

Load dataset functions

MNIST

tensorlayer.files.load_mnist_dataset(shape=(-1, 784), path='data/mnist/')[source]

Automatically download MNIST dataset and return the training, validation and test set with 50000, 10000 and 10000 digit images respectively.

Parameters:

shape : tuple

The shape of digit images, defaults is (-1,784)

path : string

The path that the data is downloaded to, defaults is data/mnist/.

Examples

>>> X_train, y_train, X_val, y_val, X_test, y_test = tl.files.load_mnist_dataset(shape=(-1,784))
>>> X_train, y_train, X_val, y_val, X_test, y_test = tl.files.load_mnist_dataset(shape=(-1, 28, 28, 1))

CIFAR-10

tensorlayer.files.load_cifar10_dataset(shape=(-1, 32, 32, 3), path='data/cifar10/', plotable=False, second=3)[source]

The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.

The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.

Parameters:

shape : tupe

The shape of digit images: e.g. (-1, 3, 32, 32) , (-1, 32, 32, 3) , (-1, 32, 32, 3)

plotable : True, False

Whether to plot some image examples.

second : int

If plotable is True, second is the display time.

path : string

The path that the data is downloaded to, defaults is data/cifar10/.

References

Examples

>>> X_train, y_train, X_test, y_test = tl.files.load_cifar10_dataset(shape=(-1, 32, 32, 3), plotable=True)

Penn TreeBank (PTB)

tensorlayer.files.load_ptb_dataset(path='data/ptb/')[source]

Penn TreeBank (PTB) dataset is used in many LANGUAGE MODELING papers, including “Empirical Evaluation and Combination of Advanced Language Modeling Techniques”, “Recurrent Neural Network Regularization”. It consists of 929k training words, 73k validation words, and 82k test words. It has 10k words in its vocabulary.

Parameters:

path : : string

The path that the data is downloaded to, defaults is data/ptb/.

Returns:

train_data, valid_data, test_data, vocabulary size

References

Examples

>>> train_data, valid_data, test_data, vocab_size = tl.files.load_ptb_dataset()

Matt Mahoney’s text8

tensorlayer.files.load_matt_mahoney_text8_dataset(path='data/mm_test8/')[source]

Download a text file from Matt Mahoney’s website if not present, and make sure it’s the right size. Extract the first file enclosed in a zip file as a list of words. This dataset can be used for Word Embedding.

Parameters:

path : : string

The path that the data is downloaded to, defaults is data/mm_test8/.

Returns:

word_list : a list

a list of string (word).

e.g. [.... ‘their’, ‘families’, ‘who’, ‘were’, ‘expelled’, ‘from’, ‘jerusalem’, ...]

Examples

>>> words = tl.files.load_matt_mahoney_text8_dataset()
>>> print('Data size', len(words))

IMBD

tensorlayer.files.load_imdb_dataset(path='data/imdb/', nb_words=None, skip_top=0, maxlen=None, test_split=0.2, seed=113, start_char=1, oov_char=2, index_from=3)[source]

Load IMDB dataset

Parameters:

path : : string

The path that the data is downloaded to, defaults is data/imdb/.

References

Examples

>>> X_train, y_train, X_test, y_test = tl.files.load_imdb_dataset(
...                                 nb_words=20000, test_split=0.2)
>>> print('X_train.shape', X_train.shape)
... (20000,)  [[1, 62, 74, ... 1033, 507, 27],[1, 60, 33, ... 13, 1053, 7]..]
>>> print('y_train.shape', y_train.shape)
... (20000,)  [1 0 0 ..., 1 0 1]

Nietzsche

tensorlayer.files.load_nietzsche_dataset(path='data/nietzsche/')[source]

Load Nietzsche dataset. Returns a string.

Parameters:

path : string

The path that the data is downloaded to, defaults is data/nietzsche/.

Examples

>>> see tutorial_generate_text.py
>>> words = tl.files.load_nietzsche_dataset()
>>> words = basic_clean_str(words)
>>> words = words.split()

English-to-French translation data from the WMT‘15 Website

tensorlayer.files.load_wmt_en_fr_dataset(path='data/wmt_en_fr/')[source]

It will download English-to-French translation data from the WMT‘15 Website (10^9-French-English corpus), and the 2013 news test from the same site as development set. Returns the directories of training data and test data.

Parameters:

path : string

The path that the data is downloaded to, defaults is data/wmt_en_fr/.

Notes

Usually, it will take a long time to download this dataset.

References

  • Code modified from /tensorflow/models/rnn/translation/data_utils.py

Flickr25k

tensorlayer.files.load_flickr25k_dataset(tag='sky', path='data/flickr25k', n_threads=50, printable=False)[source]

Returns a list of images by a given tag from Flick25k dataset, it will download Flickr25k from the official website at the first time you use it.

Parameters:

tag : string or None

If you want to get images with tag, use string like ‘dog’, ‘red’, see Flickr Search. If you want to get all images, set to None.

path : string

The path that the data is downloaded to, defaults is data/flickr25k/.

n_threads : int, number of thread to read image.

printable : bool, print infomation when reading images, default is False.

Examples

  • Get images with tag of sky
>>> images = tl.files.load_flickr25k_dataset(tag='sky')
  • Get all images
>>> images = tl.files.load_flickr25k_dataset(tag=None, n_threads=100, printable=True)

Flickr1M

tensorlayer.files.load_flickr1M_dataset(tag='sky', size=10, path='data/flickr1M', n_threads=50, printable=False)[source]

Returns a list of images by a given tag from Flickr1M dataset, it will download Flickr1M from the official website at the first time you use it.

Parameters:

tag : string or None

If you want to get images with tag, use string like ‘dog’, ‘red’, see Flickr Search. If you want to get all images, set to None.

size : int 1 to 10.

1 means 100k images ... 5 means 500k images, 10 means all 1 million images. Default is 10.

path : string

The path that the data is downloaded to, defaults is data/flickr25k/.

n_threads : int, number of thread to read image.

printable : bool, print infomation when reading images, default is False.

Examples

  • Use 200k images
>>> images = tl.files.load_flickr1M_dataset(tag='zebra', size=2)
  • Use 1 Million images
>>> images = tl.files.load_flickr1M_dataset(tag='zebra')

CycleGAN

tensorlayer.files.load_cyclegan_dataset(filename='summer2winter_yosemite', path='data/cyclegan')[source]

Load image data from CycleGAN’s database, see this link.

Parameters:

filename : string

The dataset you want, see this link.

path : string

The path that the data is downloaded to, defaults is data/cyclegan

Examples

>>> im_train_A, im_train_B, im_test_A, im_test_B = load_cyclegan_dataset(filename='summer2winter_yosemite')

Load and save network

Save network into list (npz)

tensorlayer.files.save_npz(save_list=[], name='model.npz', sess=None)[source]

Input parameters and the file name, save parameters into .npz file. Use tl.utils.load_npz() to restore.

Parameters:

save_list : a list

Parameters want to be saved.

name : a string or None

The name of the .npz file.

sess : None or Session

Notes

If you got session issues, you can change the value.eval() to value.eval(session=sess)

References

Examples

>>> tl.files.save_npz(network.all_params, name='model_test.npz', sess=sess)
... File saved to: model_test.npz
>>> load_params = tl.files.load_npz(name='model_test.npz')
... Loading param0, (784, 800)
... Loading param1, (800,)
... Loading param2, (800, 800)
... Loading param3, (800,)
... Loading param4, (800, 10)
... Loading param5, (10,)
>>> put parameters into a TensorLayer network, please see assign_params()

Load network from list (npz)

tensorlayer.files.load_npz(path='', name='model.npz')[source]

Load the parameters of a Model saved by tl.files.save_npz().

Parameters:

path : a string

Folder path to .npz file.

name : a string or None

The name of the .npz file.

Returns:

params : list

A list of parameters in order.

References

Examples

  • See save_npz and assign_params

Assign a list of parameters to network

tensorlayer.files.assign_params(sess, params, network)[source]

Assign the given parameters to the TensorLayer network.

Parameters:

sess : TensorFlow Session. Automatically run when sess is not None.

params : a list

A list of parameters in order.

network : a Layer class

The network to be assigned

Returns:

ops : list

A list of tf ops in order that assign params. Support sess.run(ops) manually.

References

Examples

>>> Save your network as follow:
>>> tl.files.save_npz(network.all_params, name='model_test.npz')
>>> network.print_params()
...
... Next time, load and assign your network as follow:
>>> tl.layers.initialize_global_variables(sess)
>>> load_params = tl.files.load_npz(name='model_test.npz')
>>> tl.files.assign_params(sess, load_params, network)
>>> network.print_params()

Load and assign a list of parameters to network

tensorlayer.files.load_and_assign_npz(sess=None, name=None, network=None)[source]

Load model from npz and assign to a network.

Parameters:

sess : TensorFlow Session

name : string

Model path.

network : a Layer class

The network to be assigned

Returns:

Returns False if faild to model is not exist.

Examples

>>> tl.files.load_and_assign_npz(sess=sess, name='net.npz', network=net)

Save network into dict (npz)

tensorlayer.files.save_npz_dict(save_list=[], name='model.npz', sess=None)[source]

Input parameters and the file name, save parameters as a dictionary into .npz file. Use tl.files.load_and_assign_npz_dict() to restore.

Parameters:

save_list : a list to tensor for parameters

Parameters want to be saved.

name : a string

The name of the .npz file.

sess : Session

Load network from dict (npz)

tensorlayer.files.load_and_assign_npz_dict(name='model.npz', sess=None)[source]

Restore the parameters saved by tl.files.save_npz_dict().

Parameters:

name : a string

The name of the .npz file.

sess : Session

Save network into ckpt

tensorlayer.files.save_ckpt(sess=None, mode_name='model.ckpt', save_dir='checkpoint', var_list=[], global_step=None, printable=False)[source]

Save parameters into ckpt file.

Parameters:

sess : Session.

mode_name : string, name of the model, default is model.ckpt.

save_dir : string, path / file directory to the ckpt, default is checkpoint.

var_list : list of variables, if not given, save all global variables.

global_step : int or None, step number.

printable : bool, if True, print all params info.

Examples

  • see tl.files.load_ckpt().

Load network from ckpt

tensorlayer.files.load_ckpt(sess=None, mode_name='model.ckpt', save_dir='checkpoint', var_list=[], is_latest=True, printable=False)[source]

Load parameters from ckpt file.

Parameters:

sess : Session.

mode_name : string, name of the model, default is model.ckpt.

Note that if is_latest is True, this function will get the mode_name automatically.

save_dir : string, path / file directory to the ckpt, default is checkpoint.

var_list : list of variables, if not given, save all global variables.

is_latest : bool, if True, load the latest ckpt, if False, load the ckpt with the name of `mode_name.

printable : bool, if True, print all params info.

Examples

  • Save all global parameters.
>>> tl.files.save_ckpt(sess=sess, mode_name='model.ckpt', save_dir='model', printable=True)
- Save specific parameters.
>>> tl.files.save_ckpt(sess=sess, mode_name='model.ckpt', var_list=net.all_params, save_dir='model', printable=True)
- Load latest ckpt.
>>> tl.files.load_ckpt(sess=sess, var_list=net.all_params, save_dir='model', printable=True)
- Load specific ckpt.
>>> tl.files.load_ckpt(sess=sess, mode_name='model.ckpt', var_list=net.all_params, save_dir='model', is_latest=False, printable=True)

Load and save variables

Save variables as .npy

tensorlayer.files.save_any_to_npy(save_dict={}, name='file.npy')[source]

Save variables to .npy file.

Examples

>>> tl.files.save_any_to_npy(save_dict={'data': ['a','b']}, name='test.npy')
>>> data = tl.files.load_npy_to_any(name='test.npy')
>>> print(data)
... {'data': ['a','b']}

Load variables from .npy

tensorlayer.files.load_npy_to_any(path='', name='file.npy')[source]

Load .npy file.

Examples

  • see save_any_to_npy()

Folder/File functions

Check file exists

tensorlayer.files.file_exists(filepath)[source]

Check whether a file exists by given file path.

Check folder exists

tensorlayer.files.folder_exists(folderpath)[source]

Check whether a folder exists by given folder path.

Delete file

tensorlayer.files.del_file(filepath)[source]

Delete a file by given file path.

Delete folder

tensorlayer.files.del_folder(folderpath)[source]

Delete a folder by given folder path.

Read file

tensorlayer.files.read_file(filepath)[source]

Read a file and return a string.

Examples

>>> data = tl.files.read_file('data.txt')

Load file list from folder

tensorlayer.files.load_file_list(path=None, regx='\\.npz', printable=True)[source]

Return a file list in a folder by given a path and regular expression.

Parameters:

path : a string or None

A folder path.

regx : a string

The regx of file name.

printable : boolean, whether to print the files infomation.

Examples

>>> file_list = tl.files.load_file_list(path=None, regx='w1pre_[0-9]+\.(npz)')

Load folder list from folder

tensorlayer.files.load_folder_list(path='')[source]

Return a folder list in a folder by given a folder path.

Parameters:

path : a string or None

A folder path.

Check and Create folder

tensorlayer.files.exists_or_mkdir(path, verbose=True)[source]

Check a folder by given name, if not exist, create the folder and return False, if directory exists, return True.

Parameters:

path : a string

A folder path.

verbose : boolean

If True, prints results, deaults is True

Returns:

True if folder exist, otherwise, returns False and create the folder

Examples

>>> tl.files.exists_or_mkdir("checkpoints/train")

Download or extract

tensorlayer.files.maybe_download_and_extract(filename, working_directory, url_source, extract=False, expected_bytes=None)[source]

Checks if file exists in working_directory otherwise tries to dowload the file, and optionally also tries to extract the file if format is ”.zip” or ”.tar”

Parameters:

filename : string

The name of the (to be) dowloaded file.

working_directory : string

A folder path to search for the file in and dowload the file to

url : string

The URL to download the file from

extract : bool, defaults is False

If True, tries to uncompress the dowloaded file is ”.tar.gz/.tar.bz2” or ”.zip” file

expected_bytes : int/None

If set tries to verify that the downloaded file is of the specified size, otherwise raises an Exception, defaults is None which corresponds to no check being performed

Returns:

filepath to dowloaded (uncompressed) file

Examples

>>> down_file = tl.files.maybe_download_and_extract(filename = 'train-images-idx3-ubyte.gz',
                                                    working_directory = 'data/',
                                                    url_source = 'http://yann.lecun.com/exdb/mnist/')
>>> tl.files.maybe_download_and_extract(filename = 'ADEChallengeData2016.zip',
                                        working_directory = 'data/',
                                        url_source = 'http://sceneparsing.csail.mit.edu/data/',
                                        extract=True)

Sort

List of string with number in human order

tensorlayer.files.natural_keys(text)[source]

Sort list of string with number in human order.

References

alist.sort(key=natural_keys) sorts in human order http://nedbatchelder.com/blog/200712/human_sorting.html (See Toothy’s implementation in the comments)

Examples

>>> l = ['im1.jpg', 'im31.jpg', 'im11.jpg', 'im21.jpg', 'im03.jpg', 'im05.jpg']
>>> l.sort(key=tl.files.natural_keys)
... ['im1.jpg', 'im03.jpg', 'im05', 'im11.jpg', 'im21.jpg', 'im31.jpg']
>>> l.sort() # that is what we dont want
... ['im03.jpg', 'im05', 'im1.jpg', 'im11.jpg', 'im21.jpg', 'im31.jpg']

Visualizing npz file

tensorlayer.files.npz_to_W_pdf(path=None, regx='w1pre_[0-9]+\\.(npz)')[source]

Convert the first weight matrix of .npz file to .pdf by using tl.visualize.W().

Parameters:

path : a string or None

A folder path to npz files.

regx : a string

Regx for the file name.

Examples

>>> Convert the first weight matrix of w1_pre...npz file to w1_pre...pdf.
>>> tl.files.npz_to_W_pdf(path='/Users/.../npz_file/', regx='w1pre_[0-9]+\.(npz)')