Confused and Scared

Hey there! It's me again. man i tell ya I need you guys help. Ok here's the story. I grew up a very nervous and scary child. (Don't ask me why). but life was scary trying new things I found to be so scary as well. I guess the unknown or maybe moving out of my comfort zone didn't work well with me. But growing up, I saw alot of things a kid shouldn't see or have to deal with in their childhood. I saw within my own home physical and domestic abuse. I saw it in my own neighborhood, but I think having in mind that I was already a scared, timid and nervous child that somehow seeing the abuse scarred me for the future. Allow me to enlighten you. I have a man that loves me with all that's in him but I am petrified of being with him and marrying him much to his hearts desire. Once in a blue moon, i feel like i want to be with him, but that swiftly changes because of the fear i carry within me. Not only that, but I'm a "Loner". I love being alone and by myself. It's all I know and I'm comfortable with it. The problem is, He's a very affectinate man and expresses his feeling alot to me. I'm not very expressive with him at all! It just feels wierd. No matter what he won't give up on me and he's been after me for nearly four years now. What do I do? and don't tell me to follow my heart people because I'm too scared to even know what my heart is telling me. This sucks! Sometimes I wish he would just go away and call it a day, but other times I ask myself am I making a mistake. what do you guys think?