The cons I find for Larger families: 1. Their lots usually have a slight (or even sometimes significant) amount of camera lag due to the increased activity. 2. Taking sims to community lots is even more of a hassle than usual. 3. It takes a longer time to play through the day, which means that a longer time needs allotted out of my personal schedule to play that family.
The cons for Smaller families: 1. There's a shortage of sims to do the work around the lot that needs done. 2. Because of everyone being so busy, social interactions are greatly decreased.

Cons/Pros for Many or Few households: None, for me. I start with a few, and let the population of households grow naturally and steadily bigger as siblings move out. Being on rotation schedule completely takes away any stress involved in too-much-choice, and not-enough-choices stress is easily relieved by the schedule's flexibility.

The pros for Larger families: 1. Tons of interpersonal interactions! There's such a big family, at least ONE sim is usually around to at least say "hi" to Suzy, and everyday SOMEsim WILL be available to teach skills, show off tricks, play pranks, dance around, or play red hands with some other sim. I hardly ever 'need' to go to a community lot because these sims are never lonely! (Which is good, because I don't quite like community lots.) 2. Because the day moves more slowly, I get to spend a lot of time with this family! I pause and play, pause and play, pause, set up some more actions, and keep an eye on some other family member, too. Multimanaging is so much fun! 3. There is hardly ever a shortage of workers to run around and make their living.
The pros for Smaller families: 1. Their lot runs very smoothly compared to the larger households. Their days go by pretty quickly, with less sims that need action-queuing. 2. Every sim usually ends up very very skilled (even though they're also very very tired and are run ragged, haha).

I personally enjoy larger families more than smaller ones, but I do want to have both! Even if the proportion is skewed towards larger. Also, my pros/cons list is made while keeping in mind my liking for Pausing and Queuing, and my current neighborhood in Test of Time challenge (which I plan to morph into the Warwickshire challenge once my little 'neandersims' destroy the Renaissance).

I prefer to play more households with fewer Sims in each. As you say, it's easier to keep track of individual Sims. It does take longer to play through a complete rotation, but individual households tend to go more quickly: if you have very big households, there's probably always somebody home and quite often somebody awake, but with small households you might have a lot of the day whiz by at triple speed while Sims are at school or at work, and again when everybody is asleep at night. I did try putting singles together early on, and having an Old Folks Home rather than playing elders as singles or couples, but I find I really prefer to have them in separate households. That way, I feel as if I'm visiting friends, not rushing through a busy subway station.

The only exception I make to that is college. Since I only instruct Sims to study when they roll a want for it, college can be very boring and playing one Sim at a time takes forever, so I quite often have a large group of Sims stay in a dorm together. I check their wants and needs when they first get up in the morning, and check individuals again if a pop-up tells me something about them, but I mostly run them on triple speed. Sims who do well and earn lots of grant money move out to private accommodation and more personal attention, but the rest just slob around in the dorms until they get kicked out (or very occasionally graduate, more by luck than judgment!). It's the only time in their lives when I'm not monitoring them pretty closely, so being in one big household gives them the freedom to develop their own interests and form relationships without adult interference.

PlatinumPlumbbob. I was referring to people who did the job THEM-SELF (for survival, not as ahobby), not hiring people, would be silly in higher ranked classes.

Mixed
There is around 30- 40 households
Some have one sim
Most have 2 - 6
A few have 7 - 12
Largest one had 16 or so.
over 170 playables at last count.

I play megahoods, and I like large families. So, the pro of a small household is that I get to know the family members better and the time flies because I never need to pause. The con is that if they are boring (Hi, I'm Jodie Larson and I just want to work all the time!) then they're boring. Whereas in a big household, if one sim is boring, someone else is interesting, and the workaholic can do her platinum happy thing in the background while someone else brings drama and excitement. Because I restart my megahoods all the time, I've gotten to know which sims are prone to boring behavior--to me, as the player, obviously, they've got their wants satisfied and their needs up so they're not bored--and can target them for inclusion in larger households. So Jodie Larson ends up managing the orphanage together with her brother. Well, she's just skilling and going to work, Jason might want a significant other but he's very low-key about it. Bring in some townie teen orphans, and the drama level goes through the roof! One of the teen girls stalks Jason (ew!) and two teens pair off and now Jodie's got to keep them from woohooing because a teen pregnancy would look really bad on her resume, and she needs to take phone calls from anyone who wants to adopt because they have to be friends with one of the orphanage managers and it's part of her job description . . .

So that's how I handle it. You bore me, you get roomies and rules of some sort, chosen to entertain me. Sims who entertain me who are in a small household can stay that way. Maybe. Depends on what they do.

In your specific situation, are you far enough along to have insane asylums? (I'm not sure about your particular play scenario.) You could always have the relatives of anysim who isn't useful to you where they are get them institutionalized and play a mini-asylum challenge inside the bigger challenge.

I like large hoods and large families. If a family is too quiet, I will do something about it - I don't mind taking a month to get through all my families, because I am in no hurry Not everyone lives in the main hood - I guess mine is kind of a custom megahood, with 4 business hoods attached to it; and I normally play the hoods in the same order, but not necessarily the families. I do get to all of them, though.

Sunbee
Well, Currently I'm re-starting my medieval hood coz I got tired of playing only (and a lot of) couples from the scratch, so I'm at the beginning.
But I'm not that fond of asylums coz I know how its played. I like to control all of my sims. If I find a sim not useful, I usually incorporate story-lines for him/her. One I use a lot is adding plagues and hospital institutes when the hood get too populated.

The Warwickshire challenge use an health system which each sim's health/life is determine on, where the a number between 0 and 115 simulate how healthy or sick they are, and they dies when it reach 0 based on various scenarios such as age, fighting, illness and pregnancies and can improved by healing/medicine from a doctor (all calculated by rolls). This how I manage my sims populations as it makes death very randomized and any life-stage except infants are affected by it. The sims who get sick still get played though.

How attached are you to your sims? By playing more, but smaller households, each individual sim will be around longer than in a 'hood with fewer households. Rotations will be longer, you'll see your sims more often as walkbys or at community lots and everyone will likely end up in everybody else's story. For this reason I prefer the 3-5 sims in a 40-50 household hood option.