Are you sure you were not seeing only notifications from your phone mirrored onto your desktop and not actual downloads? Bear in mind that is one of KDE Connect's features, showing notifications on the desktop of stuff happening on the phone. It doesn't mean that what the notification is telling you is happening on the desktop.