Not quite how I'd describe it. See if this makes sense:
A normal isolation transformer *should* not have any connection between the "ground" pins on input and output. In other words, it *should* break the ground path completely.
It sounds like, in your case, the isolation transformer did not break the ground path on the scope as expected. Although perhaps another possibility is that the transformer was set properly but something else completed the ground path. For example, maybe the scope chassis is metal and was resting on a grounded metal table or countertop, which provided a ground path. In any case, it sounds like your scope had its chassis grounded when you expected it not to be.
As for why isolating the rectifier circuit worked, even though the transformer may not be isolating the way we want it to, there are several factors here:
1) As @IBCNU stated above, in North American mains power, the neutral is tied to earth ground at the breaker panel, so electrically they're basically the same (there are important safety reasons to treat them differently and follow all applicable electrical codes, but for the purposes of the following analysis, they're the same.)
2) Your rectifier circuit used a two-prong plug, meaning it utilizes 120V Line and Neutral, but no third connection for Earth Ground.
3) Because of the Earth Ground-Neutral connection in item 1, your rectifier circuit is still ground referenced, even when it's only connected to Line and Neutral.
4) Because the rectifier circuit is ground-referenced through the neutral on the input side, you can see in @CharlesWMcDonald's simulation above that the rectified output "rides" the AC waveform. In other words, the difference between the two output terminals is rectified DC, but both output terminals still carry a high AC voltage relative to Earth Ground.
5) Assume for the sake of analysis that *your* isolation transformer is providing transformer isolation for Line and Neutral, but is providing a direct (not isolated) connection for Earth Ground.
6) For all practical purposes in terms of safety, the signal ground, the Earth Ground, and any exposed metal frame on your scope are all equally grounded.
When you add all these facts together, this is what you get:
In the first scenario, when you tried to isolate your scope, it was still grounded. When you connected your signal ground to the rectifier output, you actually connected AC mains (on the rectifier output) directly to Earth Ground (on the scope "signal" ground clip.)
In the second scenario, isolating the rectifier circuit, the transformer isolation worked, because you only have a two prong plug. The rectifier circuit was then "floating" with no DC connection to Earth Ground. When you connected the scope probes to it, that grounded the negative side of the output to Earth Ground, providing a stable reference against which to measure the positive output.
Sorry I wrote quite a lot there. Hopefully this makes sense. Let me know if you have any questions.