How do you add pandas series filling each series' missing values separately

If I have two series:

import pandas as pd
import numpy as np

s1 = pd.Series([1, np.nan, 2, np.nan])
s2 = pd.Series([1, 2, np.nan, np.nan])

I want to fill s1's missing values with 0 and s2's missing values with 1. But I want to preserve missing if both are missing.

I could:

s1.fillna(0).add(s2.fillna(1))

but this returns:

0    2.0
1    2.0
2    3.0
3    1.0
dtype: float64

And I want:

0    2.0
1    2.0
2    3.0
3    NaN
dtype: float64