Couples who live together before marriage have a higher divorce rate. That is one thing to worry about.
I think they are thinking of the statistics that say that those who live together before marriage are less likely to GET MARRIED if they are not already engaged when they begin co-habitating.
Now, onto my opinion of whether or not you should live together... my first question would be, "What is the rush?" You both have just fairly recently moved out from your familys' homes. I truly believe experiencing life all on your own is crucial before you settle down. What would you do if he cheated on you years from now? Yes, it could happen. Just read ppl's stories here and you'll see it's usually unexpected. Or if you have other unforseen problems? What then? If you don't have a clear sense of self and haven't truly been independent you won't survive when things go wrong. You'll be needy because you've always needed someone to be right there.
Most ppl who settled down when they were young later regret it. They too thought it was a good idea at the time. What tends to happen is that they later realise they were scared of being alone OR desperately needed to fill the void of the love of a parent OR they regretted never experiencing what it was like to be on their own for very long OR they realise they were too imature to have a life long commitment (end up arguing like kids throughout their entire relationship/marriage). Regret... Regret... Regret.
There aren't many success stories. Also, you may feel like you know what you want in life right now, but it will change dramatically over the next 5 years. Ask anyone 25+ year old and they'll tell you that what they knew they wanted when they were 20 totally changed. Every one of them will. That experience may be where your family is coming from. They don't want you to ruin an otherwise good relationship by rushing things. I hate to say it, but it's an immature thought to "know" you completely have the tools to decide who you want to spend the rest of your life with and how to be successful at it.. and that he does too?!
What's the rush? You have more to lose by living together right now and more to gain by waiting a few years. Love each other, grow independantly, and you will be developing the skills needed to have a long, loving relationship with yourself and your significant other.
I think deep down in your heart you may have hesitation about this and that is why you posted here. Doing this now, while you're just beginning to have a sense of self, will most likely be detrimental to your personal growth & his.