In short, no, it's not too late. The challenges you are facing are not insurmountable, even if they may seem so at the moment. Here are some helpful tips for an 'older' musician:
1) Most people have to practice much harder at your age to achieve the same results that most achieve relatively quickly when they are younger - you are not alone here. You are past the 'critical age', during which one acquires language skills more naturally, and music is a language. But keep putting in the effort, it may seem excessive but eventually it *will* pay off and result in the level of proficiency you desire. Remember, everyone runs into roadblocks when learning, no matter what their age!
2) Think positively! If slow results yield to frustration, frustration will in turn yield to hopelessness, and you will ultimately give up your dream.
3) I was in a similar situation, the only difference being I started two years younger than you. I did not develop my ear until my early twenties, when I earned my degree in music theory. I had to work much harder than my classmates, most of whom had started serious study on their instruments when they were children. If you work hard, you will overcome your weaknesses, which brings me to...
4) Your musical memory. Since you're attaching such importance to it I'm guessing it's your weakness. It is of course important, and fortunately it can be learned. However, there are several other skills that will aid you in reaching your goals. Do not dismiss your talent in these other areas! Even though you may have to work harder on your memory, try to figure out your strengths and focus on these also.
So, if you keep a positive attitude and follow the advice people have been giving you, in the end you will accomplish your goals. Just the fact that you are asking indicates that you still think it's possible, and you are correct.