I've always assumed 'When looking at the speaker'.
Though in reality, there are few true left and right speakers. If they are designated as such, it probably means the tweeter is not centered. But, it then comes a matter of do you prefer the tweeter to be off-center to the outside or to the inside. And the answer is, which ever works for you.
The only thing I consider is whether the balance control matches the speaker positions. When I turn the balance control to the right or clockwise, the speaker on the left should fade out.
Steve/bluewizard