Personally, I would say no. If you use a ton of 3rd party AU effects, you might be able to load more in the 32-bit AU bridge than in 32-bit Logic, but you're still limited by only 4GB of RAM so I can't see it making a difference.
The "64bit" thing only refers to memory addressing, not to audio engine etc...
32bit addressing means that Logic can access up to 4GB RAM. So being in 64bit mode has no advantage if you only have that much RAM.
And since the OS and other stuff also requires RAM, even 5GB total physical RAM might have little to no advantage to Logic in 64bit mode.
Furthermore, since EXS24 can see it's own RAM outside 32bitLogic's 4GB max space already, if you use a lot of EXS24 then having 5-8GB of RAM might yield little difference again booting Logic in 64bit mode, since you were already taking advantage of MORE than 4GB in total with Logic.
In fact, there are certain things that DO NOT currently work in 64bit mode (like Firewire video and Euphonix control etc etc etc) so in your case it makes no sense to even experiment with it too much.
Thank you for that excellent explanation of the 32 bit / 64 bit quagmire. It may be years before 64 bit hardware / software compatibility will be a true reality, for many of us, i suppose. It seems that hardware is generally ahead of software development.