jeudi 23 juin 2016

Memory Error in Python When Loading Dataset

I'm trying to load the Cifar-10 dataset. The code that I'm using using is from this class (I'm not enrolled though), and I keep getting told that there is a Memory Error. The code that loads it is: X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir) which is a call to: def load_CIFAR10(ROOT): """ load all of cifar """ xs = [] ys = [] for b in range(1,6): f = os.path.join(ROOT, 'data_batch_%d' % (b, )) X, Y = load_CIFAR_batch(f)//this is the important line xs.append(X) ys.append(Y) Xtr = np.concatenate(xs) Ytr = np.concatenate(ys) del X, Y Xte, Yte = load_CIFAR_batch(os.path.join(ROOT, 'test_batch')) return Xtr, Ytr, Xte, Yte which in turn calls: def load_CIFAR_batch(filename): """ load single batch of cifar """ with open(filename, 'rb') as f: datadict = pickle.load(f) X = datadict['data'] Y = datadict['labels'] X = X.reshape(10000, 3, 32, 32).transpose(0,2,3,1).astype("float")//This is the important line Y = np.array(Y) return X, Y It gets stuck on line 7 above and stops with a memory error. I looked at the memory usage using windows task manager and it was using ~800,000kb every time I ran it, no matter what other programs were running on my system. I have 8gb of RAM on my system as well. Also, in the past pyton has used gigabytes of data but on this project it won't go above 800,000kb. What's also strange is that I changed the type to "float32" and then I got the memory error at ~400,000 and it won't go higher than that. I'm using Ipython if it makes a difference. Does anyone have any ideas of what is going on or how to fix this? Edit: This also happened with around 4gb of memory free.

Aucun commentaire:

Enregistrer un commentaire