API - Files¶
Load benchmark dataset, save and restore model, save and load variables.
TensorFlow provides .ckpt
file format to save and restore the models, while
we suggest to use standard python file format .npz
to save models for the
sake of cross-platform.
# save model as .ckpt
saver = tf.train.Saver()
save_path = saver.save(sess, "model.ckpt")
# restore model from .ckpt
saver = tf.train.Saver()
saver.restore(sess, "model.ckpt")
# save model as .npz
tl.files.save_npz(network.all_params , name='model.npz')
# restore model from .npz
load_params = tl.files.load_npz(path='', name='model.npz')
tl.files.assign_params(sess, load_params, network)
# you can assign the pre-trained parameters as follow
# 1st parameter
tl.files.assign_params(sess, [load_params[0]], network)
# the first three parameters
tl.files.assign_params(sess, load_params[:3], network)
load_mnist_dataset ([shape, path]) |
Automatically download MNIST dataset and return the training, validation and test set with 50000, 10000 and 10000 digit images respectively. |
load_cifar10_dataset ([shape, path, …]) |
The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. |
load_ptb_dataset ([path]) |
Penn TreeBank (PTB) dataset is used in many LANGUAGE MODELING papers, including “Empirical Evaluation and Combination of Advanced Language Modeling Techniques”, “Recurrent Neural Network Regularization”. |
load_matt_mahoney_text8_dataset ([path]) |
Download a text file from Matt Mahoney’s website if not present, and make sure it’s the right size. |
load_imdb_dataset ([path, nb_words, …]) |
Load IMDB dataset |
load_nietzsche_dataset ([path]) |
Load Nietzsche dataset. |
load_wmt_en_fr_dataset ([path]) |
It will download English-to-French translation data from the WMT‘15 Website (10^9-French-English corpus), and the 2013 news test from the same site as development set. |
load_flickr25k_dataset ([tag, path, …]) |
Returns a list of images by a given tag from Flick25k dataset, it will download Flickr25k from the official website at the first time you use it. |
load_flickr1M_dataset ([tag, size, path, …]) |
Returns a list of images by a given tag from Flickr1M dataset, it will download Flickr1M from the official website at the first time you use it. |
load_cyclegan_dataset ([filename, path]) |
Load image data from CycleGAN’s database, see this link. |
save_npz ([save_list, name, sess]) |
Input parameters and the file name, save parameters into .npz file. |
load_npz ([path, name]) |
Load the parameters of a Model saved by tl.files.save_npz(). |
assign_params (sess, params, network) |
Assign the given parameters to the TensorLayer network. |
load_and_assign_npz ([sess, name, network]) |
Load model from npz and assign to a network. |
save_npz_dict ([save_list, name, sess]) |
Input parameters and the file name, save parameters as a dictionary into .npz file. |
load_and_assign_npz_dict ([name, sess]) |
Restore the parameters saved by tl.files.save_npz_dict() . |
save_ckpt ([sess, mode_name, save_dir, …]) |
Save parameters into ckpt file. |
load_ckpt ([sess, mode_name, save_dir, …]) |
Load parameters from ckpt file. |
save_any_to_npy ([save_dict, name]) |
Save variables to .npy file. |
load_npy_to_any ([path, name]) |
Load .npy file. |
file_exists (filepath) |
Check whether a file exists by given file path. |
folder_exists (folderpath) |
Check whether a folder exists by given folder path. |
del_file (filepath) |
Delete a file by given file path. |
del_folder (folderpath) |
Delete a folder by given folder path. |
read_file (filepath) |
Read a file and return a string. |
load_file_list ([path, regx, printable]) |
Return a file list in a folder by given a path and regular expression. |
load_folder_list ([path]) |
Return a folder list in a folder by given a folder path. |
exists_or_mkdir (path[, verbose]) |
Check a folder by given name, if not exist, create the folder and return False, if directory exists, return True. |
maybe_download_and_extract (filename, …[, …]) |
Checks if file exists in working_directory otherwise tries to dowload the file, and optionally also tries to extract the file if format is “.zip” or “.tar” |
natural_keys (text) |
Sort list of string with number in human order. |
npz_to_W_pdf ([path, regx]) |
Convert the first weight matrix of .npz file to .pdf by using tl.visualize.W(). |
Load dataset functions¶
MNIST¶
-
tensorlayer.files.
load_mnist_dataset
(shape=(-1, 784), path='data/mnist/')[source]¶ Automatically download MNIST dataset and return the training, validation and test set with 50000, 10000 and 10000 digit images respectively.
Parameters: - shape : tuple
The shape of digit images, defaults is (-1,784)
- path : string
The path that the data is downloaded to, defaults is
data/mnist/
.
Examples
>>> X_train, y_train, X_val, y_val, X_test, y_test = tl.files.load_mnist_dataset(shape=(-1,784)) >>> X_train, y_train, X_val, y_val, X_test, y_test = tl.files.load_mnist_dataset(shape=(-1, 28, 28, 1))
CIFAR-10¶
-
tensorlayer.files.
load_cifar10_dataset
(shape=(-1, 32, 32, 3), path='data/cifar10/', plotable=False, second=3)[source]¶ The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.
The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.
Parameters: - shape : tupe
The shape of digit images: e.g. (-1, 3, 32, 32) , (-1, 32, 32, 3) , (-1, 32, 32, 3)
- plotable : True, False
Whether to plot some image examples.
- second : int
If
plotable
is True,second
is the display time.- path : string
The path that the data is downloaded to, defaults is
data/cifar10/
.
References
Examples
>>> X_train, y_train, X_test, y_test = tl.files.load_cifar10_dataset(shape=(-1, 32, 32, 3), plotable=True)
Penn TreeBank (PTB)¶
-
tensorlayer.files.
load_ptb_dataset
(path='data/ptb/')[source]¶ Penn TreeBank (PTB) dataset is used in many LANGUAGE MODELING papers, including “Empirical Evaluation and Combination of Advanced Language Modeling Techniques”, “Recurrent Neural Network Regularization”. It consists of 929k training words, 73k validation words, and 82k test words. It has 10k words in its vocabulary.
Parameters: - path : : string
The path that the data is downloaded to, defaults is
data/ptb/
.
Returns: - train_data, valid_data, test_data, vocabulary size
References
tensorflow.models.rnn.ptb import reader
- Manual download
Examples
>>> train_data, valid_data, test_data, vocab_size = tl.files.load_ptb_dataset()
Matt Mahoney’s text8¶
-
tensorlayer.files.
load_matt_mahoney_text8_dataset
(path='data/mm_test8/')[source]¶ Download a text file from Matt Mahoney’s website if not present, and make sure it’s the right size. Extract the first file enclosed in a zip file as a list of words. This dataset can be used for Word Embedding.
Parameters: - path : : string
The path that the data is downloaded to, defaults is
data/mm_test8/
.
Returns: - word_list : a list
a list of string (word).
e.g. […. ‘their’, ‘families’, ‘who’, ‘were’, ‘expelled’, ‘from’, ‘jerusalem’, …]
Examples
>>> words = tl.files.load_matt_mahoney_text8_dataset() >>> print('Data size', len(words))
IMBD¶
-
tensorlayer.files.
load_imdb_dataset
(path='data/imdb/', nb_words=None, skip_top=0, maxlen=None, test_split=0.2, seed=113, start_char=1, oov_char=2, index_from=3)[source]¶ Load IMDB dataset
Parameters: - path : : string
The path that the data is downloaded to, defaults is
data/imdb/
.
References
Examples
>>> X_train, y_train, X_test, y_test = tl.files.load_imdb_dataset( ... nb_words=20000, test_split=0.2) >>> print('X_train.shape', X_train.shape) ... (20000,) [[1, 62, 74, ... 1033, 507, 27],[1, 60, 33, ... 13, 1053, 7]..] >>> print('y_train.shape', y_train.shape) ... (20000,) [1 0 0 ..., 1 0 1]
Nietzsche¶
-
tensorlayer.files.
load_nietzsche_dataset
(path='data/nietzsche/')[source]¶ Load Nietzsche dataset. Returns a string.
Parameters: - path : string
The path that the data is downloaded to, defaults is
data/nietzsche/
.
Examples
>>> see tutorial_generate_text.py >>> words = tl.files.load_nietzsche_dataset() >>> words = basic_clean_str(words) >>> words = words.split()
English-to-French translation data from the WMT‘15 Website¶
-
tensorlayer.files.
load_wmt_en_fr_dataset
(path='data/wmt_en_fr/')[source]¶ It will download English-to-French translation data from the WMT‘15 Website (10^9-French-English corpus), and the 2013 news test from the same site as development set. Returns the directories of training data and test data.
Parameters: - path : string
The path that the data is downloaded to, defaults is
data/wmt_en_fr/
.
Notes
Usually, it will take a long time to download this dataset.
References
- Code modified from /tensorflow/models/rnn/translation/data_utils.py
Flickr25k¶
-
tensorlayer.files.
load_flickr25k_dataset
(tag='sky', path='data/flickr25k', n_threads=50, printable=False)[source]¶ Returns a list of images by a given tag from Flick25k dataset, it will download Flickr25k from the official website at the first time you use it.
Parameters: - tag : string or None
If you want to get images with tag, use string like ‘dog’, ‘red’, see Flickr Search. If you want to get all images, set to
None
.- path : string
The path that the data is downloaded to, defaults is
data/flickr25k/
.- n_threads : int, number of thread to read image.
- printable : bool, print infomation when reading images, default is
False
.
Examples
- Get images with tag of sky
>>> images = tl.files.load_flickr25k_dataset(tag='sky')
- Get all images
>>> images = tl.files.load_flickr25k_dataset(tag=None, n_threads=100, printable=True)
Flickr1M¶
-
tensorlayer.files.
load_flickr1M_dataset
(tag='sky', size=10, path='data/flickr1M', n_threads=50, printable=False)[source]¶ Returns a list of images by a given tag from Flickr1M dataset, it will download Flickr1M from the official website at the first time you use it.
Parameters: - tag : string or None
If you want to get images with tag, use string like ‘dog’, ‘red’, see Flickr Search. If you want to get all images, set to
None
.- size : int 1 to 10.
1 means 100k images … 5 means 500k images, 10 means all 1 million images. Default is 10.
- path : string
The path that the data is downloaded to, defaults is
data/flickr25k/
.- n_threads : int, number of thread to read image.
- printable : bool, print infomation when reading images, default is
False
.
Examples
- Use 200k images
>>> images = tl.files.load_flickr1M_dataset(tag='zebra', size=2)
- Use 1 Million images
>>> images = tl.files.load_flickr1M_dataset(tag='zebra')
CycleGAN¶
-
tensorlayer.files.
load_cyclegan_dataset
(filename='summer2winter_yosemite', path='data/cyclegan')[source]¶ Load image data from CycleGAN’s database, see this link.
Parameters: - filename : string
The dataset you want, see this link.
- path : string
The path that the data is downloaded to, defaults is data/cyclegan
Examples
>>> im_train_A, im_train_B, im_test_A, im_test_B = load_cyclegan_dataset(filename='summer2winter_yosemite')
Load and save network¶
Save network into list (npz)¶
-
tensorlayer.files.
save_npz
(save_list=[], name='model.npz', sess=None)[source]¶ Input parameters and the file name, save parameters into .npz file. Use tl.utils.load_npz() to restore.
Parameters: - save_list : a list
Parameters want to be saved.
- name : a string or None
The name of the .npz file.
- sess : None or Session
Notes
If you got session issues, you can change the value.eval() to value.eval(session=sess)
References
Examples
>>> tl.files.save_npz(network.all_params, name='model_test.npz', sess=sess) ... File saved to: model_test.npz >>> load_params = tl.files.load_npz(name='model_test.npz') ... Loading param0, (784, 800) ... Loading param1, (800,) ... Loading param2, (800, 800) ... Loading param3, (800,) ... Loading param4, (800, 10) ... Loading param5, (10,) >>> put parameters into a TensorLayer network, please see assign_params()
Load network from list (npz)¶
-
tensorlayer.files.
load_npz
(path='', name='model.npz')[source]¶ Load the parameters of a Model saved by tl.files.save_npz().
Parameters: - path : a string
Folder path to .npz file.
- name : a string or None
The name of the .npz file.
Returns: - params : list
A list of parameters in order.
References
Examples
- See save_npz and assign_params
Assign a list of parameters to network¶
-
tensorlayer.files.
assign_params
(sess, params, network)[source]¶ Assign the given parameters to the TensorLayer network.
Parameters: - sess : TensorFlow Session. Automatically run when sess is not None.
- params : a list
A list of parameters in order.
- network : a
Layer
class The network to be assigned
Returns: - ops : list
A list of tf ops in order that assign params. Support sess.run(ops) manually.
References
Examples
>>> Save your network as follow: >>> tl.files.save_npz(network.all_params, name='model_test.npz') >>> network.print_params() ... ... Next time, load and assign your network as follow: >>> tl.layers.initialize_global_variables(sess) >>> load_params = tl.files.load_npz(name='model_test.npz') >>> tl.files.assign_params(sess, load_params, network) >>> network.print_params()
Load and assign a list of parameters to network¶
-
tensorlayer.files.
load_and_assign_npz
(sess=None, name=None, network=None)[source]¶ Load model from npz and assign to a network.
Parameters: - sess : TensorFlow Session
- name : string
Model path.
- network : a
Layer
class The network to be assigned
Returns: - Returns False if faild to model is not exist.
Examples
>>> tl.files.load_and_assign_npz(sess=sess, name='net.npz', network=net)
Save network into dict (npz)¶
-
tensorlayer.files.
save_npz_dict
(save_list=[], name='model.npz', sess=None)[source]¶ Input parameters and the file name, save parameters as a dictionary into .npz file. Use
tl.files.load_and_assign_npz_dict()
to restore.Parameters: - save_list : a list to tensor for parameters
Parameters want to be saved.
- name : a string
The name of the .npz file.
- sess : Session
Load network from dict (npz)¶
Save network into ckpt¶
-
tensorlayer.files.
save_ckpt
(sess=None, mode_name='model.ckpt', save_dir='checkpoint', var_list=[], global_step=None, printable=False)[source]¶ Save parameters into ckpt file.
Parameters: - sess : Session.
- mode_name : string, name of the model, default is
model.ckpt
. - save_dir : string, path / file directory to the ckpt, default is
checkpoint
. - var_list : list of variables, if not given, save all global variables.
- global_step : int or None, step number.
- printable : bool, if True, print all params info.
Examples
- see
tl.files.load_ckpt()
.
Load network from ckpt¶
-
tensorlayer.files.
load_ckpt
(sess=None, mode_name='model.ckpt', save_dir='checkpoint', var_list=[], is_latest=True, printable=False)[source]¶ Load parameters from ckpt file.
Parameters: - sess : Session.
- mode_name : string, name of the model, default is
model.ckpt
. Note that if
is_latest
is True, this function will get themode_name
automatically.- save_dir : string, path / file directory to the ckpt, default is
checkpoint
. - var_list : list of variables, if not given, save all global variables.
- is_latest : bool, if True, load the latest ckpt, if False, load the ckpt with the name of
`mode_name
. - printable : bool, if True, print all params info.
Examples
- Save all global parameters.
>>> tl.files.save_ckpt(sess=sess, mode_name='model.ckpt', save_dir='model', printable=True) - Save specific parameters. >>> tl.files.save_ckpt(sess=sess, mode_name='model.ckpt', var_list=net.all_params, save_dir='model', printable=True) - Load latest ckpt. >>> tl.files.load_ckpt(sess=sess, var_list=net.all_params, save_dir='model', printable=True) - Load specific ckpt. >>> tl.files.load_ckpt(sess=sess, mode_name='model.ckpt', var_list=net.all_params, save_dir='model', is_latest=False, printable=True)
Folder/File functions¶
Check file exists¶
Check folder exists¶
Delete folder¶
Read file¶
Load file list from folder¶
-
tensorlayer.files.
load_file_list
(path=None, regx='\\.npz', printable=True)[source]¶ Return a file list in a folder by given a path and regular expression.
Parameters: - path : a string or None
A folder path.
- regx : a string
The regx of file name.
- printable : boolean, whether to print the files infomation.
Examples
>>> file_list = tl.files.load_file_list(path=None, regx='w1pre_[0-9]+\.(npz)')
Load folder list from folder¶
Check and Create folder¶
-
tensorlayer.files.
exists_or_mkdir
(path, verbose=True)[source]¶ Check a folder by given name, if not exist, create the folder and return False, if directory exists, return True.
Parameters: - path : a string
A folder path.
- verbose : boolean
If True, prints results, deaults is True
Returns: - True if folder exist, otherwise, returns False and create the folder
Examples
>>> tl.files.exists_or_mkdir("checkpoints/train")
Download or extract¶
-
tensorlayer.files.
maybe_download_and_extract
(filename, working_directory, url_source, extract=False, expected_bytes=None)[source]¶ Checks if file exists in working_directory otherwise tries to dowload the file, and optionally also tries to extract the file if format is “.zip” or “.tar”
Parameters: - filename : string
The name of the (to be) dowloaded file.
- working_directory : string
A folder path to search for the file in and dowload the file to
- url : string
The URL to download the file from
- extract : bool, defaults is False
If True, tries to uncompress the dowloaded file is “.tar.gz/.tar.bz2” or “.zip” file
- expected_bytes : int/None
If set tries to verify that the downloaded file is of the specified size, otherwise raises an Exception, defaults is None which corresponds to no check being performed
Returns: - filepath to dowloaded (uncompressed) file
Examples
>>> down_file = tl.files.maybe_download_and_extract(filename = 'train-images-idx3-ubyte.gz', working_directory = 'data/', url_source = 'http://yann.lecun.com/exdb/mnist/') >>> tl.files.maybe_download_and_extract(filename = 'ADEChallengeData2016.zip', working_directory = 'data/', url_source = 'http://sceneparsing.csail.mit.edu/data/', extract=True)
Sort¶
List of string with number in human order¶
-
tensorlayer.files.
natural_keys
(text)[source]¶ Sort list of string with number in human order.
References
alist.sort(key=natural_keys) sorts in human order http://nedbatchelder.com/blog/200712/human_sorting.html (See Toothy’s implementation in the comments)
Examples
>>> l = ['im1.jpg', 'im31.jpg', 'im11.jpg', 'im21.jpg', 'im03.jpg', 'im05.jpg'] >>> l.sort(key=tl.files.natural_keys) ... ['im1.jpg', 'im03.jpg', 'im05', 'im11.jpg', 'im21.jpg', 'im31.jpg'] >>> l.sort() # that is what we dont want ... ['im03.jpg', 'im05', 'im1.jpg', 'im11.jpg', 'im21.jpg', 'im31.jpg']
Visualizing npz file¶
-
tensorlayer.files.
npz_to_W_pdf
(path=None, regx='w1pre_[0-9]+\\.(npz)')[source]¶ Convert the first weight matrix of .npz file to .pdf by using tl.visualize.W().
Parameters: - path : a string or None
A folder path to npz files.
- regx : a string
Regx for the file name.
Examples
>>> Convert the first weight matrix of w1_pre...npz file to w1_pre...pdf. >>> tl.files.npz_to_W_pdf(path='/Users/.../npz_file/', regx='w1pre_[0-9]+\.(npz)')