isnan | StackOverflow

### Answer rating: 197

I have an array of floats (some normal numbers, some nans) that is coming out of an apply on a pandas dataframe.

For some reason, numpy.isnan is failing on this array, however as shown below, each element is a float, numpy.isnan runs correctly on each element, the type of the variable is definitely a numpy array.

What"s going on?!

```
set([type(x) for x in tester])
Out[59]: {float}
tester
Out[60]:
array([-0.7000000000000001, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan, nan], dtype=object)
set([type(x) for x in tester])
Out[61]: {float}
np.isnan(tester)
Traceback (most recent call last):
File "<ipython-input-62-e3638605b43c>", line 1, in <module>
np.isnan(tester)
TypeError: ufunc "isnan" not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ""safe""
set([np.isnan(x) for x in tester])
Out[65]: {False, True}
type(tester)
Out[66]: numpy.ndarray
```

`np.isnan`

can be applied to NumPy arrays of native dtype (such as np.float64):

```
In [99]: np.isnan(np.array([np.nan, 0], dtype=np.float64))
Out[99]: array([ True, False], dtype=bool)
```

but raises TypeError when applied to object arrays:

```
In [96]: np.isnan(np.array([np.nan, 0], dtype=object))
TypeError: ufunc "isnan" not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ""safe""
```

Since you have Pandas, you could use `pd.isnull`

instead -- it can accept NumPy arrays of object or native dtypes:

```
In [97]: pd.isnull(np.array([np.nan, 0], dtype=float))
Out[97]: array([ True, False], dtype=bool)
In [98]: pd.isnull(np.array([np.nan, 0], dtype=object))
Out[98]: array([ True, False], dtype=bool)
```

Note that `None`

is also considered a null value in object arrays.

I have an array of floats (some normal numbers, some nans) that is coming out of an apply on a pandas dataframe.

For some reason, numpy.isnan is failing on this array, however as shown below, each element is a float, numpy.isnan runs correctly on each element, the type of the variable is definitely a numpy array.

What"s going on?!

```
set([type(x) for x in tester])
Out[59]: {float}
tester
Out[60]:
array([-0.7000000000000001, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan, nan], dtype=object)
set([type(x) for x in tester])
Out[61]: {float}
np.isnan(tester)
Traceback (most recent call last):
File "<ipython-input-62-e3638605b43c>", line 1, in <module>
np.isnan(tester)
TypeError: ufunc "isnan" not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ""safe""
set([np.isnan(x) for x in tester])
Out[65]: {False, True}
type(tester)
Out[66]: numpy.ndarray
```

`np.isnan`

can be applied to NumPy arrays of native dtype (such as np.float64):

```
In [99]: np.isnan(np.array([np.nan, 0], dtype=np.float64))
Out[99]: array([ True, False], dtype=bool)
```

but raises TypeError when applied to object arrays:

```
In [96]: np.isnan(np.array([np.nan, 0], dtype=object))
TypeError: ufunc "isnan" not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ""safe""
```

Since you have Pandas, you could use `pd.isnull`

instead -- it can accept NumPy arrays of object or native dtypes:

```
In [97]: pd.isnull(np.array([np.nan, 0], dtype=float))
Out[97]: array([ True, False], dtype=bool)
In [98]: pd.isnull(np.array([np.nan, 0], dtype=object))
Out[98]: array([ True, False], dtype=bool)
```

Note that `None`

is also considered a null value in object arrays.

There are lots of things I have seen make a model diverge.

Too high of a learning rate. You can often tell if this is the case if the loss begins to increase and then diverges to infinity.

I am not to familiar with the DNNClassifier but I am guessing it uses the categorical cross entropy cost function. This involves taking the log of the prediction which diverges as the prediction approaches zero. That is why people usually add a small epsilon value to the prediction to prevent this divergence. I am guessing the DNNClassifier probably does this or uses the tensorflow opp for it. Probably not the issue.

Other numerical stability issues can exist such as division by zero where adding the epsilon can help. Another less obvious one if the square root who"s derivative can diverge if not properly simplified when dealing with finite precision numbers. Yet again I doubt this is the issue in the case of the DNNClassifier.

You may have an issue with the input data. Try calling

`assert not np.any(np.isnan(x))`

on the input data to make sure you are not introducing the nan. Also make sure all of the target values are valid. Finally, make sure the data is properly normalized. You probably want to have the pixels in the range [-1, 1] and not [0, 255].The labels must be in the domain of the loss function, so if using a logarithmic-based loss function all labels must be non-negative (as noted by evan pu and the comments below).

```
import pandas as pd
import numpy as np
import math
# For single variable all three libraries return single boolean
x1 = float("nan")
print(f"It"s pd.isna: {pd.isna(x1)}")
print(f"It"s np.isnan: {np.isnan(x1)}}")
print(f"It"s math.isnan: {math.isnan(x1)}}")
```

**Output**

```
It"s pd.isna: True
It"s np.isnan: True
It"s math.isnan: True
```

This might happen inside scikit, and it depends on what you"re doing. I recommend reading the documentation for the functions you"re using. You might be using one which depends e.g. on your matrix being positive definite and not fulfilling that criteria.

**EDIT**: How could I miss that:

```
np.isnan(mat.any()) #and gets False
np.isfinite(mat.all()) #and gets True
```

is obviously wrong. Right would be:

```
np.any(np.isnan(mat))
```

and

```
np.all(np.isfinite(mat))
```

You want to check wheter any of the element is NaN, and not whether the return value of the `any`

function is a number...

Return

`True`

if x is a NaN (not a number), and`False`

otherwise.

```
>>> import math
>>> x = float("nan")
>>> math.isnan(x)
True
```

The usual way to test for a NaN is to see if it"s equal to itself:

```
def isNaN(num):
return num != num
```

If you"re using numpy for your arrays, you can also use

```
x = x[numpy.logical_not(numpy.isnan(x))]
```

Equivalently

```
x = x[~numpy.isnan(x)]
```

[Thanks to chbrown for the added shorthand]

**Explanation**

The inner function, `numpy.isnan`

returns a boolean/logical array which has the value `True`

everywhere that `x`

is not-a-number. As we want the opposite, we use the logical-not operator, `~`

to get an array with `True`

s everywhere that `x`

**is** a valid number.

Lastly we use this logical array to index into the original array `x`

, to retrieve just the non-NaN values.

This should be faster than iterating and will work regardless of shape.

```
numpy.isnan(myarray).any()
```

Edit: 30x faster:

```
import timeit
s = "import numpy;a = numpy.arange(10000.).reshape((100,100));a[10,10]=numpy.nan"
ms = [
"numpy.isnan(a).any()",
"any(numpy.isnan(x) for x in a.flatten())"]
for m in ms:
print " %.2f s" % timeit.Timer(m, s).timeit(1000), m
```

Results:

```
0.11 s numpy.isnan(a).any()
3.75 s any(numpy.isnan(x) for x in a.flatten())
```

Bonus: it works fine for non-array NumPy types:

```
>>> a = numpy.float64(42.)
>>> numpy.isnan(a).any()
False
>>> a = numpy.float64(numpy.nan)
>>> numpy.isnan(a).any()
True
```

`numpy.isnan(number)`

tells you if it"s `NaN`

or not.

```
np.count_nonzero(~np.isnan(data))
```

`~`

inverts the boolean matrix returned from `np.isnan`

.

`np.count_nonzero`

counts values that is not 0false. `.sum`

should give the same result. But maybe more clearly to use `count_nonzero`

Testing speed:

```
In [23]: data = np.random.random((10000,10000))
In [24]: data[[np.random.random_integers(0,10000, 100)],:][:, [np.random.random_integers(0,99, 100)]] = np.nan
In [25]: %timeit data.size - np.count_nonzero(np.isnan(data))
1 loops, best of 3: 309 ms per loop
In [26]: %timeit np.count_nonzero(~np.isnan(data))
1 loops, best of 3: 345 ms per loop
In [27]: %timeit data.size - np.isnan(data).sum()
1 loops, best of 3: 339 ms per loop
```

`data.size - np.count_nonzero(np.isnan(data))`

seems to barely be the fastest here. other data might give different relative speed results.

X
# Submit new EBook