dimanche 26 juin 2016

KeyError while using UnbalancedDataset package to over-sample a dataset (in pandas.index.IndexEngine.get_loc)

I am trying to use UnbalancedDataset to over-sample my data. Following the sklearn convention, I have X,y as the feature matrix and target vector. These are of the pandas.core.frame.DataFrame type with a shape of (200000, 17) and (200000,) respectively.

I first split the data using the sklean's train_test_split. Then applied the SMOTE method to over-sample the training dataset which resulted in the following error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
C:Users...Anaconda3libsite-packagespandasindexesbase.py in get_loc(self, key, method, tolerance)
   1944             try:
-> 1945                 return self._engine.get_loc(key)
   1946             except KeyError:

pandasindex.pyx in pandas.index.IndexEngine.get_loc (pandasindex.c:4154)()

pandasindex.pyx in pandas.index.IndexEngine.get_loc (pandasindex.c:4018)()

pandashashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandashashtable.c:12368)()

pandashashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandashashtable.c:12322)()

KeyError: 1143

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-99-1c5830417b3f> in <module>()
      6 # 'SMOTE'
      7 SM = SMOTE(ratio=ratio, verbose=verbose, kind='regular')
----> 8 smx, smy = SM.fit_transform(Xtrain, ytrain)

C:Users...Anaconda3libsite-packagesunbalanceddataset-0.1-py3.5.eggunbalanced_datasetunbalanced_dataset.py in fit_transform(self, x, y)
    274             return self.out_x, self.out_y, self.out_idx
    275         else:
--> 276             self.out_x, self.out_y = self.resample()
    277 
    278             return self.out_x, self.out_y

C:Users...Anaconda3libsite-packagesunbalanceddataset-0.1-py3.5.eggunbalanced_datasetover_sampling.py in resample(self)
    358                                        step_size=1.0,
    359                                        random_state=self.rs,
--> 360                                        verbose=self.verbose)
    361 
    362             if self.verbose:

C:Users...Anaconda3libsite-packagesunbalanceddataset-0.1-py3.5.eggunbalanced_datasetunbalanced_dataset.py in make_samples(x, nn_data, y_type, nn_num, n_samples, step_size, random_state, verbose)
    388 
    389             # Construct synthetic sample
--> 390             new[i] = x[row] - step * (x[row] - nn_data[nn_num[row, col]])
    391 
    392         # The returned target vector is simply a repetition of the

C:Users...Anaconda3libsite-packagespandascoreframe.py in __getitem__(self, key)
   1995             return self._getitem_multilevel(key)
   1996         else:
-> 1997             return self._getitem_column(key)
   1998 
   1999     def _getitem_column(self, key):

C:Users...Anaconda3libsite-packagespandascoreframe.py in _getitem_column(self, key)
   2002         # get column
   2003         if self.columns.is_unique:
-> 2004             return self._get_item_cache(key)
   2005 
   2006         # duplicate columns & possible reduce dimensionality

C:Users...Anaconda3libsite-packagespandascoregeneric.py in _get_item_cache(self, item)
   1348         res = cache.get(item)
   1349         if res is None:
-> 1350             values = self._data.get(item)
   1351             res = self._box_item_values(item, values)
   1352             cache[item] = res

C:Users...Anaconda3libsite-packagespandascoreinternals.py in get(self, item, fastpath)
   3288 
   3289             if not isnull(item):
-> 3290                 loc = self.items.get_loc(item)
   3291             else:
   3292                 indexer = np.arange(len(self.items))[isnull(self.items)]

C:Users...Anaconda3libsite-packagespandasindexesbase.py in get_loc(self, key, method, tolerance)
   1945                 return self._engine.get_loc(key)
   1946             except KeyError:
-> 1947                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   1948 
   1949         indexer = self.get_indexer([key], method=method, tolerance=tolerance)

pandasindex.pyx in pandas.index.IndexEngine.get_loc (pandasindex.c:4154)()

pandasindex.pyx in pandas.index.IndexEngine.get_loc (pandasindex.c:4018)()

pandashashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandashashtable.c:12368)()

pandashashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandashashtable.c:12322)()

KeyError: 1143

I got this error while all the under-sampling methods of the UnbalancedDataset on the same data worked fine. Any suggestions to handle the problem with oversampling?

UPDATE:

To address the issue, the Pandas DataFrame needs to be converted to Numpy matrix from. So, the following conversion would solve the problem:

Xc = Xtrain.as_matrix()

Aucun commentaire:

Enregistrer un commentaire