As far as where to locate, obviously there are benefits to both uptown and downtown, but you have to think where your clients are coming from. If someone is going to make a trip especially to see a girl, i.e. travel for 30 min or so, then either location works. However, if you're looking for people who find you online, or tipped through a review board like here, or business travellers in a hotel downtown, that's where you have to be, I think. If you stay downtown you likely won't lose any customers, but if you leave you might.
I agree with the guy who warned you about the hours and about the condo owners - you've got to have the right setup, don't forget LE might actually bother you if you're an incall - not likely, but possible, and more so if there's a complaint from a neighbour!
Have you thought about the buildings at Bay/Charles, other buildings downtown? Friends used to be at Yonge/College, Roommates is still there I think?