Status update July 4th
Just wanted to let you know where we are with Lemmy.world.
Issues
As you might have noticed, things still won’t work as desired… we see several issues:
Performance
- Loading is mostly OK, but sometimes things take forever
- We (and you) see many 502 errors, resulting in empty pages etc.
- System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)
Bugs
- Replying to a DM doesn’t seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
- 2FA seems to be a problem for many people. It doesn’t always work as expected.
Troubleshooting
We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @nutomic@lemmy.ml is also helping with current issues.
So, all is not yet running smoothly as we hoped, but with all this help we’ll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!
Thanks again for all the hard work on Lemmy World. It feels fast today
This is the level of transparency that most companies should strive for. Ironic that in terms of fixing things, volunteer and passion projects seem to be more on top of issues compared to big companies with hundreds of employees.
You said it: passion projects. While being paid is surely a motivator, seeing your pet project take off the way Lemmy is can be so intoxicating and rewarding! I plan to donate as soon as I get paid on Friday! I want to see this succeed, even if it is just to spite Reddit, and I am willing to pay for the pleasure.
i just wanted to thank you for doing your best to fix lemmy.world as soon as possible.
but please, don’t feel forced to overwork yourselves. i understand you want to do it soon so more people can move from Reddit, but i wouldn’t like that Lemmy software and community developers overwork and feel miserable, as those things are some of the very motives you escaped from Reddit in first place.
in my opinion, it would be nice that we users understand this situation and, if we want lemmy so bad, we actively help with it.
this applies to all lemmy instances and communities, ofc. have a nice day you all! ^^
Plus, slow steady growth means eventual success. Burnout is very real if you never take a break.
Thanks for all of your effort. Even though we are on different instances, it’s important for the Fediverse community that you succeed. You are doing valuable work, and I appreciate it.
Not just that, but the code contributed to Lemmy by this debugging will make Lemmy run faster for everyone on every instance, which is makes the ecosystem that much better.
Huge respect for what you’ve built here, but it might be worth reaching out to the lemm.ee admin. I only know enough DevOps and cloud hosting to be dangerous, not helpful. But his instance seems stable and scalable. He might be able to offer some insight into the issues here
Yes he’s one of the other admins in our Discord, he’s very helpful!
That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%
Lemmy has a memory leak? Or, should I say, a “lemmory leak”?
Lemmy has a memory leak? Or, should I say, a “lemmory leak”?
A lemmory meek, obviously!
Wait isn’t lemmy written in rust how do you create a memory leak in rust? Unsafe mode?
let mut memory_leak = Vec::<u8>::new(); loop { memory_leak.push(0); }That’s not a memory leak though. That’s just hording memory. Leaked memory is inaccessible.
In the example it’s inaccessible.
It’s not. The vec is still accessible and if it goes out of scope rust automatically clears the memory.
But it will never fall out of scope because of the loop
Exactly.
Exactly but it’s still accessible since it’s in scope.
I’m just excited to be back in the Wild West again – all of the big players had bumps, at least this one is working to fix them.
I want this to succeed so badly. I truly feel like it’s going to be sink or swim and will reflect how all enshitification efforts will play out.
Band together now and people see there’s a chance. Fail and we are doomed to corporate greed in every facet of our lives.
Wow. Drama! Chill bruh. They’ll get there. If it takes too long for some, they can switch to other instances.
I’m here for the drama too!
Fuck the cleanliness of the corporate web.
More props for you and the other admins as you ride the waves. It’s for a great cause.
.world is definitely running smoother than when I joined 3 days ago, back then it was impossible to comment and the lag was immense, right now I just have to occasionally reload the page, but that’s nothing in comparison.
You guys are doing an amazing work! I’m broke, so here are some
coins 🪙🪙🪙🪙beans 🫘🫘🫘🫘good news, a fix might be in the works: https://github.com/LemmyNet/lemmy/pull/3482
Yes phiresky is working with us improving performance, has helped a lot so far! (We’re now running a custom build with some of his improvements)
I’ve noticed lemmy.world has been much more responsive today so something seems to be working!
deleted by creator
o7
Respect to all lemmy.world admins!
The need to restart server every so often to avoid excessive ram usage bit is very interesting to me. This sounds like some issue with memory management. Not necessarily a leak, but maybe something like server keeping unnecessary references so the object cannot be dropped.
Anyway, from my experience Rust developers love debugging such kind of problems. Are Lemmy Devs aware of this issue? And do you publish server usage logs somewhere to look deeper into that?
server keeping unnecessary references so the object cannot be dropped
You indeed just described a memory leak :D
Are there any plans to make user up and down votes not viewable publicly? I know for a lot of new adopters this can be a deal breaker.
If someone cares about their privacy that badly why vote at all? I’m sure votes are tracked by the sites on other websites?
On reddit only reddit knows your up and down votes which are never made public unless you check an option in settings. The fediverse is already a target for brigading due to the decentralized nature, allowing bad actors to figure out who to target seems like a terrible idea. Imagine what people like /r/againsthatesubreddits would do with that info.
On the flip side, it also makes brigading easier to detect, so instances can defederate communities that routinely vote in bad faith.
So only group think supported brigading is allowed, even better.
The redirects to baseball games glitch is gone. I am unsure if this is because the glitch is fixed or if baseball season is over.
😆 sorry about that! Baseball season is still very much on.
This issue was caused by the websockets replacing the current thread you were on with newly updated or created posts. Our game threads get updated frequently so they were constantly popping up in people’s faces. We decreased the frequency of the updates pretty substantially in order to be less annoying to unrelated instances but it was still happening occasionally.
Now that we’re all through with websockets it won’t be an issue anymore!
