Was my brother out of line?

Tell your brother you appreciate that his comments are coming out of love and concern for you, but your relationships with the other people in your life are yours to deal with, and you'd appreciate it if he confines future conversations to the relationship that you and he share instead of offering opinions that aren't based on complete information. That, or open up to your family about the reasons behind the separation - though if they have a habit of being judgmental, I can see why you'd be reluctant to do so.