Don't compare with the other. The main point is you two and the baby.
If really have financal problem and need to discuss with both parents, then speak clearly. If they don't understand and then give a bad feedback to both of you. Then will you think this is a good family member? A bad family member - no meeting with them any more is not a problem. Although you may care other people will say you two are bad to parent, will it affect your daily life?
If the thing you do is right, then no need to care other people's point of view.