Edit: obligatory explanation (thanks mods for squaring me away)…
What you see via the UI isn’t “all that exists”. Unlike Reddit, where everything is a black box, there are a lot more eyeballs who can see “under the hood”. Any instance admin, proper or rogue, gets a ton of information that users won’t normally see. The attached example demonstrates that while users will only see upvote/downvote tallies, admins can see who actually performed those actions.
Edit: Obligatory RIP my inbox.
Another option would be to aggregate votes per instance so programming.dev might only see “42 upvotes from lemmy.world”, but not user details. I don’t think that changes the inter-instance trust equation, at least not notably, and it even works in conjunction with non-aggregated upvotes and displaying everything publicly.