That reminds me...
It is my understanding that the majority of Chinese immigrants into North America prior to 1900 spoke Cantonese. Mostly because the main port of exit in coming to America in the period was Hong Kong. Shanghai became an important portal to the Americas in the late 19th century, bringing more Mandarin speakers to places like San Francisco and Vancouver.
Having no direct links to Chinese or Chinese-American culture, I don't know how accurate this is. Anyone know?
I also am not an expert with Chinese history. I only know a bit of the national tongue. From my own speculation, what you stated sounds very plausible as there would be a lot of disenfranchised Cantonese as a result of the opium wars with the British and subsequent fallout, looking for a way to make a living and survive.
I also suspect that a good number in southern China would be under the grips of the last dynasty, the Ming dynasty. This would be the decades before the resentment towards the Western world broke out and the Boxer Rebellion of the early 1900s so that would also be a time of great poverty and want with large numbers looking to immigrate and find a way to survive.
Either way, those Chinese most likely would represent the poorer sections of the populations, the former farmers, the uneducated and unskilled laborers who would find themselves without jobs, and other types like this.
Like I said, I literally have no idea what I'm really saying, just going off a basic college education and some guesswork so any authority on the matter is free to refute me with fear or repercussion or hit to my ego.