Short term you should be fine. Long term you will very likely have problems.
Some people have pulled it off but it really depends on the fish. If you get a fairly aggressive pair, they will likely not tolerate the others.
Make sure you have quite a few spearate structures. Also making high piles of rocks, plants or driftwood to help break up sight lines will help a lot.
Do you have any dither/target fish in there?