Adding an attribute to input children of hidden elements with jquery?

I'm trying to accomplish something with JQuery and I can't seem to get it to work. A lot of that has to do with me not being a JQuery guy, so I'm sure this is all wrong.

I have a very large form, with some conditionals to where certain form sections don't show up, depending on what radio button value is selected. So what I've done, is when a certain value is selected, it will add or remove a "hide-me" class to a div element, so all form fields inside don't show. The issue is, if I have some required, the form won't submit and you can't fill out a field that isn't visible. So what I want to do is when the form is submitted, it should first grab all child input elements within any div with the class of hide-me and set an attribute of disabled.

How can I accomplish this?

Here is some example code: http://codepen.io/amarit84/pen/KMgKRd

$('#test_form_id').submit(function() {
    $('.hide-me').children(":input").prop("disabled", true);
});