I don't know where exactly in the city/long island you're at but you might want to try a place called country critters. It's inbetween sunrise hwy and montauk hwy off of rte. 112 south. Not sure what town it's in, exactly. This place is amazing, though. They are set up like an actual aquarium (almost like mystic seaport in connecticut). They have beautiful tanks and a huge variety. The staff is also EXTREMELY knowledgeable. It's a great place to walk around in even if you're not buying anything. Some of the setups they have are just absolutely amazing. They could probably get away with charging admission.
There's also Magic Isle. They have a very good staff and very good selection as well. They're in Centereach in the Walmart shopping center off of Mark Tree Rd. That's where I get my fish from. They're pretty good there. They're also not big on selling you crap that you don't need. One of the employees even told me not to get a filter from them as they are much cheaper on-line (mostly because on-line wholesalers don't really have as much overhead as local businesses).
You can also check out this other place in Lindenhurst. Not sure what it's called. It's like an aquarium wharehouse. Not as nice on the inside but they have a lot to pick from at good prices. If you take rte 110 south just past sunrise hwy, you'll see it on the right hand side of the road. I believe the sign just says "Pets" or something like that. You'd have to keep an eye out for it as it's kind of hard to spot. I'd say it's no more than 5 minutes from the intersection of 110 and sunrise (27).
Depending on where you are, you could also try a place called Aqua Hut. It's a bit of a dive but they have really low prices and a large selection. Just be very cautious about the fish you get from there. I'd definitely use a qt tank for any of these. But, if you get a healthy fish it'll be at a great price so it might be worth it. I'm not saying their fish look diseased. On the contrary. Most of them appear healthy. They're just not housed in great conditions, which could cause unseen problems. This is off of rte. 112 as well, heading south. It's in a home depot shopping center near the intersection of rte 25 and 112 (kind of a confusing intersection, so be carfeful if you're heading that way).
Those are the fish stores within a 45 minute radius of me. The one closest to NYC would be the aquarium wharehouse place. If you're in the city, it still might be a little bit of a haul for you. I think all of these stores are pretty much located in the middle portion of LI. Hope that helps.