Well, of course, that is your choice. You may listen in any manner that pleases you.
The works themselves were intended to be played as a unit, and were key-related, and followed a general tempo pattern (if we're talking symphonies here) of fast, slow, dance, and very fast. So from that standpoint, the work is not complete until you traveled through the keys and tempos. Now, as far as motivic unity between the movements -- that is a development that came rather later in the Classical period. An early example would be Mozart's Jupiter Symphony, in which a 4 note motive -- C-D-F-E -- is heard in all 4 movements. And it becomes the main theme in the finale.
In Beethoven's 5th Symphony the rhythmic "3 shorts and 1 long" idea is used in 3 of the 4 movements.
Of course there is the famous "Idee Fix" of Berlioz' "Synfonie Fantastique". A rather extended melodic idea is heard in all 5 movements.
Composers were subtle, and did not always wish to smack you in the face with what their unifying idea was -- so listening alone may not reveal it -- you might have to have a smattering of musical knowledge and maybe a guidebook at times -- but the curious thing is that although we don't know what the systematic unifying framework might be, we migh notice that the various movements SOUND related in a way that we can't quite identify, but it is there neverhteless.
Happy listening!!