Edit: obligatory explanation (thanks mods for squaring me away)…
What you see via the UI isn’t “all that exists”. Unlike Reddit, where everything is a black box, there are a lot more eyeballs who can see “under the hood”. Any instance admin, proper or rogue, gets a ton of information that users won’t normally see. The attached example demonstrates that while users will only see upvote/downvote tallies, admins can see who actually performed those actions.
Edit: Obligatory RIP my inbox.
Except it is not just your instance and the instance hosting the post (which would make sense). It is every instance in the federation. So if I want to see how you vote, I just need to pay few bucks for hosting and make my own instance.
Actually, it’s even easier than that, just make a kbin account then go to any Lemmy instance and you can check the upvotes/downvotes.