Droughts are becoming increasingly common in India, where 50% of the labor force works in agriculture, and most agricultural production is rainfall dependent. This paper investigates the extent to which rural households adapt to drought by reallocating labor from agriculture to other sectors of the economy. I use household-level fixed effects regressions to estimate to find that household agricultural jobs decline in the year following a drought. Further, I find that these effects are mediated by job skills and land ownership. I find that households with working members who have completed primary education account for most of the workers who exit the agricultural sector. In contrast, I find that households that own land increase their agricultural labor share after experiencing a drought. Thus, while I find that drought causes households to diversify away from agriculture on aggregate, the extent of this structural change is mitigated by the behavior of landowners.