The S3 is just a fun to drive as the S4. S4 was a bit bigger on the inside, but I still feel comfortable in the S3. As far as handling, I can tell the S4 was a bigger car when throwing around curves. The S3 handling is much more nimble than the S4 and more fun in the corners. As far as power, the S4 (with the supercharger) was much more linear whereas the S3's turbo punch, you feel the power more. Off the line I know the S3 is quicker (not by much), but the S4 is no slouch either.
I was looking for a smaller car and started to look at the S3. Even a CPO S4 was in the price range of a nicely equipped S3 so I went with it.
Not regrets as I love all the option in the Prestige model and love the color (Sepang) and the sport seats remind me of the seats in the S4.
If Audi brings the S3 hatchback to the US, you could carry your dogs in it. I would agree with you, the S3 and CRV are light years apart, that's a CUV and the S3 is a sports sedan. Getting used to the S3's size didn't take a huge amount of time, or not owning a Honda for the last nine years I got used to it easily. It was a switch that I don't regret ever making. It is my VERY FIRST all wheel drive car, even though some people complain about the Haldex system used in the car. My dad's 2015 Q50 has all wheel drive, and their system is really cool. The S3 is very usable and practical.