Who Thinks That I am Being Selfish?

I'm in my early twenties and I recently broke up with my boyfriend of over 4 years. The new man that I am with is in his early thirties. My ex boyfriend and I used to spend a lot of time together when we were both off of work and school. We saw each other every day since we got together for over 4 years. But things got to where we barely ever did anything fun anymore because we did not have enough time for each other. Leading to our break up. Now with this new man we both usually have conflicting work schedules, but recently I have been working mornings and have more free time in the evenings to go places with him. And conveniently this week he's been making a ton of excuses not to spend time with me when he gets off in the evening. And so far I only saw him for one night this week. He works, (can't be mad at that, understandable, everyone has to work) he also goes to the gym after work and his football season is over that used to take up much of his time. Now being with someone else for over four years, both of us working and going to school full time, and still trying to find time to do something to me was way more difficult. This man is done with school and is just working right now. I am on a break from school for the summer and just working. I think that it should be easier to spend time together and do things like go to the movies. But he cancelled the movies at least 2 times this week saying he was dealing with some personal issues involving his mom, brothers and sisters. When I tried to tell him that I feel like he could make more time for me he tells me I'm being selfish. And that i need to be more vocal about when i want to do something with him but I always tell him that I want to see him and he blows me off. He always acts like I should just show up at his apt in the evenings (since that's when we can see each other) but he's always so busy i call and check first to make sure he's free.I tell him that if i am coming to stay at his place he should be telling me when he's available for me. It is common courtesy to ask someone before coming to see them if they are ready for company. He blames me for not having time for him when i have had time for him but he chooses to do other things when he knows i want to see him. Am I being selfish by wanting to spend time with him? He's my boyfriend. I just don't think that its right to be in a "relationship" with someone and see each other 1 or 2 times a week when you could see each other more.