API - Load, Save Model and Data¶
Load benchmark dataset, save and restore model, save and load variables.
TensorFlow provides .ckpt
file format to save and restore the models, while
we suggest to use standard python file format .npz
to save models for the
sake of cross-platform.
# save model as .ckpt
saver = tf.train.Saver()
save_path = saver.save(sess, "model.ckpt")
# restore model from .ckpt
saver = tf.train.Saver()
saver.restore(sess, "model.ckpt")
# save model as .npz
tl.files.save_npz(network.all_params , name='model.npz')
# restore model from .npz
load_params = tl.files.load_npz(path='', name='model.npz')
tl.files.assign_params(sess, load_params, network)
# you can assign the pre-trained parameters as follow
# 1st parameter
tl.files.assign_params(sess, [load_params[0]], network)
# the first three parameters
tl.files.assign_params(sess, load_params[:3], network)
load_mnist_dataset ([shape, path]) |
Automatically download MNIST dataset and return the training, validation and test set with 50000, 10000 and 10000 digit images respectively. |
load_cifar10_dataset ([shape, path, …]) |
The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. |
load_ptb_dataset ([path]) |
Penn TreeBank (PTB) dataset is used in many LANGUAGE MODELING papers, including “Empirical Evaluation and Combination of Advanced Language Modeling Techniques”, “Recurrent Neural Network Regularization”. |
load_matt_mahoney_text8_dataset ([path]) |
Download a text file from Matt Mahoney’s website if not present, and make sure it’s the right size. |
load_imbd_dataset |
|
load_nietzsche_dataset ([path]) |
Load Nietzsche dataset. |
load_wmt_en_fr_dataset ([path]) |
It will download English-to-French translation data from the WMT‘15 Website (10^9-French-English corpus), and the 2013 news test from the same site as development set. |
save_npz ([save_list, name, sess]) |
Input parameters and the file name, save parameters into .npz file. |
save_npz_dict ([save_list, name, sess]) |
Input parameters and the file name, save parameters as a dictionary into .npz file. |
load_npz ([path, name]) |
Load the parameters of a Model saved by tl.files.save_npz(). |
load_npz_dict ([path, name]) |
Load the parameters of a Model saved by tl.files.save_npz_dict(). |
assign_params (sess, params, network) |
Assign the given parameters to the TensorLayer network. |
load_and_assign_npz ([sess, name, network]) |
Load model from npz and assign to a network. |
save_any_to_npy ([save_dict, name]) |
Save variables to .npy file. |
load_npy_to_any ([path, name]) |
Load .npy file. |
npz_to_W_pdf ([path, regx]) |
Convert the first weight matrix of .npz file to .pdf by using tl.visualize.W(). |
load_file_list ([path, regx, printable]) |
Return a file list in a folder by given a path and regular expression. |
load_folder_list ([path]) |
Return a folder list in a folder by given a folder path. |
exists_or_mkdir (path[, verbose]) |
Check a folder by given name, if not exist, create the folder and return False, if directory exists, return True. |
maybe_download_and_extract (filename, …[, …]) |
Checks if file exists in working_directory otherwise tries to dowload the file, and optionally also tries to extract the file if format is “.zip” or “.tar” |
Load dataset functions¶
MNIST¶
-
tensorlayer.files.
load_mnist_dataset
(shape=(-1, 784), path='data/mnist/')[source]¶ Automatically download MNIST dataset and return the training, validation and test set with 50000, 10000 and 10000 digit images respectively.
Parameters: - shape : tuple
The shape of digit images, defaults to (-1,784)
- path : string
Path to download data to, defaults to data/mnist/
Examples
>>> X_train, y_train, X_val, y_val, X_test, y_test = tl.files.load_mnist_dataset(shape=(-1,784)) >>> X_train, y_train, X_val, y_val, X_test, y_test = tl.files.load_mnist_dataset(shape=(-1, 28, 28, 1))
CIFAR-10¶
-
tensorlayer.files.
load_cifar10_dataset
(shape=(-1, 32, 32, 3), path='data/cifar10/', plotable=False, second=3)[source]¶ The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.
The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.
Parameters: - shape : tupe
The shape of digit images: e.g. (-1, 3, 32, 32) , (-1, 32, 32, 3) , (-1, 32*32*3)
- plotable : True, False
Whether to plot some image examples.
- second : int
If
plotable
is True,second
is the display time.- path : string
Path to download data to, defaults to data/cifar10/
Notes
CIFAR-10 images can only be display without color change under uint8. >>> X_train = np.asarray(X_train, dtype=np.uint8) >>> plt.ion() >>> fig = plt.figure(1232) >>> count = 1 >>> for row in range(10): >>> for col in range(10): >>> a = fig.add_subplot(10, 10, count) >>> plt.imshow(X_train[count-1], interpolation=’nearest’) >>> plt.gca().xaxis.set_major_locator(plt.NullLocator()) # 不显示刻度(tick) >>> plt.gca().yaxis.set_major_locator(plt.NullLocator()) >>> count = count + 1 >>> plt.draw() >>> plt.pause(3)
References
Examples
>>> X_train, y_train, X_test, y_test = tl.files.load_cifar10_dataset(shape=(-1, 32, 32, 3), plotable=True)
Penn TreeBank (PTB)¶
-
tensorlayer.files.
load_ptb_dataset
(path='data/ptb/')[source]¶ Penn TreeBank (PTB) dataset is used in many LANGUAGE MODELING papers, including “Empirical Evaluation and Combination of Advanced Language Modeling Techniques”, “Recurrent Neural Network Regularization”.
It consists of 929k training words, 73k validation words, and 82k test words. It has 10k words in its vocabulary.
In “Recurrent Neural Network Regularization”, they trained regularized LSTMs of two sizes; these are denoted the medium LSTM and large LSTM. Both LSTMs have two layers and are unrolled for 35 steps. They initialize the hidden states to zero. They then use the final hidden states of the current minibatch as the initial hidden state of the subsequent minibatch (successive minibatches sequentially traverse the training set). The size of each minibatch is 20.
The medium LSTM has 650 units per layer and its parameters are initialized uniformly in [−0.05, 0.05]. They apply 50% dropout on the non-recurrent connections. They train the LSTM for 39 epochs with a learning rate of 1, and after 6 epochs they decrease it by a factor of 1.2 after each epoch. They clip the norm of the gradients (normalized by minibatch size) at 5.
The large LSTM has 1500 units per layer and its parameters are initialized uniformly in [−0.04, 0.04]. We apply 65% dropout on the non-recurrent connections. They train the model for 55 epochs with a learning rate of 1; after 14 epochs they start to reduce the learning rate by a factor of 1.15 after each epoch. They clip the norm of the gradients (normalized by minibatch size) at 10.
Parameters: - path : : string
Path to download data to, defaults to data/ptb/
Returns: - train_data, valid_data, test_data, vocabulary size
Examples
>>> train_data, valid_data, test_data, vocab_size = tl.files.load_ptb_dataset()
Matt Mahoney’s text8¶
-
tensorlayer.files.
load_matt_mahoney_text8_dataset
(path='data/mm_test8/')[source]¶ Download a text file from Matt Mahoney’s website if not present, and make sure it’s the right size. Extract the first file enclosed in a zip file as a list of words. This dataset can be used for Word Embedding.
Parameters: - path : : string
Path to download data to, defaults to data/mm_test8/
Returns: - word_list : a list
a list of string (word).
e.g. […. ‘their’, ‘families’, ‘who’, ‘were’, ‘expelled’, ‘from’, ‘jerusalem’, …]
Examples
>>> words = tl.files.load_matt_mahoney_text8_dataset() >>> print('Data size', len(words))
IMBD¶
Nietzsche¶
-
tensorlayer.files.
load_nietzsche_dataset
(path='data/nietzsche/')[source]¶ Load Nietzsche dataset. Returns a string.
Parameters: - path : string
Path to download data to, defaults to data/nietzsche/
Examples
>>> see tutorial_generate_text.py >>> words = tl.files.load_nietzsche_dataset() >>> words = basic_clean_str(words) >>> words = words.split()
English-to-French translation data from the WMT‘15 Website¶
-
tensorlayer.files.
load_wmt_en_fr_dataset
(path='data/wmt_en_fr/')[source]¶ It will download English-to-French translation data from the WMT‘15 Website (10^9-French-English corpus), and the 2013 news test from the same site as development set. Returns the directories of training data and test data.
Parameters: - path : string
Path to download data to, defaults to data/wmt_en_fr/
Notes
Usually, it will take a long time to download this dataset.
References
- Code modified from /tensorflow/models/rnn/translation/data_utils.py
Load and save network¶
Save network into list (npz)¶
-
tensorlayer.files.
save_npz
(save_list=[], name='model.npz', sess=None)[source]¶ Input parameters and the file name, save parameters into .npz file. Use tl.utils.load_npz() to restore.
Parameters: - save_list : a list
Parameters want to be saved.
- name : a string or None
The name of the .npz file.
- sess : None or Session
Notes
If you got session issues, you can change the value.eval() to value.eval(session=sess)
References
Examples
>>> tl.files.save_npz(network.all_params, name='model_test.npz', sess=sess) ... File saved to: model_test.npz >>> load_params = tl.files.load_npz(name='model_test.npz') ... Loading param0, (784, 800) ... Loading param1, (800,) ... Loading param2, (800, 800) ... Loading param3, (800,) ... Loading param4, (800, 10) ... Loading param5, (10,) >>> put parameters into a TensorLayer network, please see assign_params()
Save network into dict (npz)¶
-
tensorlayer.files.
save_npz_dict
(save_list=[], name='model.npz', sess=None)[source]¶ Input parameters and the file name, save parameters as a dictionary into .npz file. Use tl.utils.load_npz_dict() to restore.
Parameters: - save_list : a list
Parameters want to be saved.
- name : a string or None
The name of the .npz file.
- sess : None or Session
Notes
This function tries to avoid a potential broadcasting error raised by numpy.
Load network from save_npz¶
-
tensorlayer.files.
load_npz
(path='', name='model.npz')[source]¶ Load the parameters of a Model saved by tl.files.save_npz().
Parameters: - path : a string
Folder path to .npz file.
- name : a string or None
The name of the .npz file.
Returns: - params : list
A list of parameters in order.
References
Examples
- See save_npz and assign_params
Load network from save_npz_dict¶
Assign parameters to network¶
-
tensorlayer.files.
assign_params
(sess, params, network)[source]¶ Assign the given parameters to the TensorLayer network.
Parameters: - sess : TensorFlow Session. Automatically run when sess is not None.
- params : a list
A list of parameters in order.
- network : a
Layer
class The network to be assigned
Returns: - ops : list
A list of tf ops in order that assign params. Support sess.run(ops) manually.
References
Examples
>>> Save your network as follow: >>> tl.files.save_npz(network.all_params, name='model_test.npz') >>> network.print_params() ... ... Next time, load and assign your network as follow: >>> tl.layers.initialize_global_variables(sess) >>> load_params = tl.files.load_npz(name='model_test.npz') >>> tl.files.assign_params(sess, load_params, network) >>> network.print_params()
Load and assign parameters to network¶
-
tensorlayer.files.
load_and_assign_npz
(sess=None, name=None, network=None)[source]¶ Load model from npz and assign to a network.
Parameters: - sess : TensorFlow Session
- name : string
Model path.
- network : a
Layer
class The network to be assigned
Returns: - Returns False if faild to model is not exist.
Examples
>>> tl.files.load_and_assign_npz(sess=sess, name='net.npz', network=net)
Visualizing npz file¶
-
tensorlayer.files.
npz_to_W_pdf
(path=None, regx='w1pre_[0-9]+\\.(npz)')[source]¶ Convert the first weight matrix of .npz file to .pdf by using tl.visualize.W().
Parameters: - path : a string or None
A folder path to npz files.
- regx : a string
Regx for the file name.
Examples
>>> Convert the first weight matrix of w1_pre...npz file to w1_pre...pdf. >>> tl.files.npz_to_W_pdf(path='/Users/.../npz_file/', regx='w1pre_[0-9]+\.(npz)')
Folder functions¶
Load file list from folder¶
-
tensorlayer.files.
load_file_list
(path=None, regx='\\.npz', printable=True)[source]¶ Return a file list in a folder by given a path and regular expression.
Parameters: - path : a string or None
A folder path.
- regx : a string
The regx of file name.
- printable : boolean, whether to print the files infomation.
Examples
>>> file_list = tl.files.load_file_list(path=None, regx='w1pre_[0-9]+\.(npz)')
Load folder list from folder¶
Check and Create folder¶
-
tensorlayer.files.
exists_or_mkdir
(path, verbose=True)[source]¶ Check a folder by given name, if not exist, create the folder and return False, if directory exists, return True.
Parameters: - path : a string
A folder path.
- verbose : boolean
If True, prints results, deaults is True
Returns: - True if folder exist, otherwise, returns False and create the folder
Examples
>>> tl.files.exists_or_mkdir("checkpoints/train")
Download or extract¶
-
tensorlayer.files.
maybe_download_and_extract
(filename, working_directory, url_source, extract=False, expected_bytes=None)[source]¶ Checks if file exists in working_directory otherwise tries to dowload the file, and optionally also tries to extract the file if format is “.zip” or “.tar”
Parameters: - filename : string
The name of the (to be) dowloaded file.
- working_directory : string
A folder path to search for the file in and dowload the file to
- url : string
The URL to download the file from
- extract : bool, defaults to False
If True, tries to uncompress the dowloaded file is “.tar.gz/.tar.bz2” or “.zip” file
- expected_bytes : int/None
If set tries to verify that the downloaded file is of the specified size, otherwise raises an Exception, defaults to None which corresponds to no check being performed
- Returns
- ———-
- filepath to dowloaded (uncompressed) file
Examples
>>> down_file = tl.files.maybe_download_and_extract(filename = 'train-images-idx3-ubyte.gz', working_directory = 'data/', url_source = 'http://yann.lecun.com/exdb/mnist/') >>> tl.files.maybe_download_and_extract(filename = 'ADEChallengeData2016.zip', working_directory = 'data/', url_source = 'http://sceneparsing.csail.mit.edu/data/', extract=True)