Actually, not super dumb...it's a feature! The text for the connector has no, uh, bounding box, so there is no true background, and, no padding.
Text has two types of background, behind the text (which I suppose is text background), and then background for the text box. Try experiment, adding text to the page. Add fill color. Now, edit the text via format...textblock, and choose a fill color, other than the background color you just added. Look at the shape, two colors! The 1st fills the text box, the second just, backlights the text. Changing the left right margin spacing does change the padding for the textbox fill, but has no impact upon text background.
Basically, what you have is what you get. You could make a custom connector, add a box for background, but, it's a lot of extra work.
HTH
Wapperdude