Exactly! That initial deceleration is pretty hard which is why jumpers try to have the plane going as slow as possible when they take that first step.You instantly create enough drag to slow down your horizontal speed, that's why jumpers are more afraid of hitting the plane they just jumped out of then hitting the ground.
The decrease in velocity in going horizontal is almost as quick as the increase in velocity as you drop vertically. After the first 10 or so seconds your horizontal travel is largely determined by the relative wind speeds and has little bearing relating to what direction or speed the plane was going.
I don't jump either but I had a college class with someone who did and he gave a super presentation on skydiving for one of our assignments.