I have had my sim travel to the other neighborhood in order to harvest different wild plants to add to her ever expanding garden. Every time I go, only the plants available in her own neighborhood, daisies and crysanthemums, are available to harvest. The lemon tree, cherry tree, blackberry bush, and lily plants can never be harvested. However, the sage plant she can harvest. Her gardening skill is level 8, so I don't think that's an issue. I even had my sim and her whole family stay the day to give time for the plants to grow in game time. The plants did grow, and some became harvestable, just not the plants I wanted her to get.

Different plants grow at different rates. I'm actually doing the same thing in my game at the moment. Trying to get a lily to harvest. You have a couple options. At level 7 (i think) you can take a cutting of the plant you want to be grafted later. The other option is to stay in that neighborhood until the plant is ready for harvest. The plants will grow while you are in the neighborhood, but you may need to wait as long as 2 or 3 days. In that case what I do is visit a house in the neighborhood and take care of any needs there, cooking, eating, sleeping while I wait for the plant to be ready for harvest.

the neighborhoods are dormant unless you hang out in them.
Plants will not grow unless you stay there for so many hours..
even food will not spoil if you leave it in a park unless you spend time in that park.
(also different plants grow in different worlds and different areas, so you may need to explore a few more)

Certain plants take longer to mature and bloom for harvest. I restarted a new save and since I just finished the 1st week, I don't have much to harvest for my gardening sims either.

Yes, I noticed they are dormant, which was why I had my sims go there for a whole day, and then returned another day to stay for another long while. I was hoping it was long enough. When they still didn't grow, I thought there was something I was missing. Apparently what I am missing is patience.

I usually take my sim to a public place such as a park, so they have access to bathrooms, food, and a place to nap.

Whoa thats a huge design flaw

Wow. It's bad enough to have to deal with a loading screen to get there in the first place. I want a crossword puzzle magazine for my sims. They are going to have to wait a while to get their plants, it seems.

It does ruin immersion a little bit when you come back to a park and notice an EXCELLENT plate of hamburgers left by you a week ago.
Must have been from Micky D's I guess.