There’s an unavoidable trade-off when choosing a firearm and ammo caliber for self-defense in and around your home. The more powerful options are also louder and have greater recoil.
Indoors, the noise is greater from any firearm due to the enclosed space. The sound bounces back from walls and ceiling, increasing the possible harm. For home defense, it’s a good idea to keep a pair of high NRR rated ear muffs with your firearm. Throw it on if you need to grab your gun for defensive purposes. But even then, it’s probably impractical for everyone in the home to have hearing protection available at all times. So you should take this factor into account when choosing a home defense gun. A .223 AR is far too loud for indoor use, especially without a silencer or hearing protection. A 12 gauge is similarly problematic.
Recoil is another significant factor. If you are an experienced shooter, fine; you know which caliber you can shoot accurately, with a quick follow-up shot. But many preppers, who are not experienced shooters, would like to own a firearm for self-defense in the home. They need a gun that does not require a lot of range time to use effectively. And so recoil is an important consideration.
If your firearm of choice for home defense is a handgun, one good lower noise lower recoil option is a revolver loaded with .38 special ammo. Muzzle energy is in the 200+ ft lb range, which is significantly less than even a 9mm standard pressure round, indicating low recoil. Bullets weights range from 90 to 158 grains. As the shooter gains more experience, they can switch to a +P loading, with greater velocity. I would also suggest choosing a revolver rated for .357 magnum ammo, with a 4 to 6 inch barrel (and reasonably high gun weight). The heavier gun further reduces recoil when shooting the .38 special ammo. The longer barrel reduces noise somewhat. And if you ever feel the need, you can switch to .357 magnum ammo.
The .22LR round has low noise and low recoil, but it is not effective enough for self-defense. Still a gun in .22LR is better than no gun at all. If you had to choose the .22 LR for defensive purposes, a rifle with 40 grain high velocity ammo is probably the best option.
For semi-auto handguns, some shooters prefer the noise and recoil of a .45 ACP to a 9mm. Technically, the .45 is louder and has more recoil. But the recoil is a more of a push, rather than a jolt. And the noise for a subsonic round is more of a dull thud than a crack. All of this is rather subjective. For lowest recoil and noise, the .380 ACP is the better option. Choose an FMJ round for greatest penetration. The hollow points expand but with very limited penetration. Again, a heavier gun and a longer barrel, within reason, will also decrease noise and recoil.
Here’s where the low noise low recoil options really shine. A pistol-caliber carbine has very low recoil, due to the heavier weight of the gun and the fact that your shoulder absorbs the recoil. Even a new shooter has no problem with the recoil of a long gun in a modest caliber. Accuracy is much greater for any shooter with a long gun than a handgun. And the longer barrel means much lower noise, as long as you choose a suitable caliber.
A lever-action rifle capable of shooting both .38 special and .357 magnum ammo is a formidable home defense gun. Lever action firearms are not subject to the same legal restrictions as ARs in certain states or localities. A .38 special round in a long gun is very soft shooting and relatively low noise. You should still throw on a pair of ear muffs if at all possible. But an inexperienced shooter can be very effective with a modest level of training with this type of gun.
Then if you feel the need or desire, you can use .357 magnum ammo in the same lever-action rifle, for top-notch defensive effectiveness. A .357 round at 158 grains and over 1500 fps is a one-shot bad-guy stopper. Outdoors, the range of this firearm is much greater than any handgun, with good terminal performance. In a revolver, the .357 is very loud and has high recoil. In a rifle, the same ammo offers very manageable recoil with lower noise.
In a semi-auto carbine, any pistol round (9mm, .40, .45) is lower noise and lower recoil. A 124 grain 9mm +P round from a 16-inch barrel has similar ballistics to a .357 round from a revolver. Very effective. There are several good 9mm carbines available on the high end of the price range. On the low end, the Hi-Point 9mm carbine is very affordable (and rather ugly). If you should desire even less recoil and noise (I’m not sure why), Hi-Point has just announced the same carbine will soon be available in .380 ACP. Ballistics are similar to a 9mm handgun.
What are your options for an AR with limited noise? I would suggest the .300 blackout with subsonic ammo is the better option. You’ve got more noise than some of the other options above, but adding a suppressor might be an option (in some states). Outdoors, you can switch to the supersonic ammo (around 110 grains) which is more effective than the .223 AR.
As always, this is just my non-expert opinion. Your mileage may vary.