500 errors

changed milestone to %Remaining issues after gitlab migration

Unfortunately I can't follow up here today.

when reporting 500 errors to OSU, include your external IP, the date/time, and the request ID that shows on the 500 page

so they can find it in their logs. also probably worth waiting until you have a few of them collected before sending them a mail

if somebody else from the infra team could do that in the meantime, it would be great. otherwise I'll try to do it over the next days.

I opened a new ticket (#33630) with OSUOSL, and include a few 500 errors that I hit earlier.

the gitlab instance was rebooted and this seems to have "fixed" it for now.

I told OSU that they could close the ticket, but I'm kinda think we should keep our issue open though. If this happens again, we can start to collect more info about it in one place (this issue). There's an upstream bug about this but it seems stalled: https://gitlab.com/gitlab-org/gitlab/-/issues/480905

added infrastructure label

added status::confirmed type::bug labels

(deleted a comment, I think posting your own IPs here publicly is not a good idea. Thanks though!)

Request ID: 01JCQA4TAWVTPJN3HDYPKYESZ pmaports!5785 (merged) 11/15 07:00 +/-3min UTC

Request ID: 01JCQGWH6YPFZPB3M86TGK86BG pmaports!5694 11/15 09:00-09:02 UTC

gitlab was restarted on Friday to work around it. (it is still https://gitlab.com/gitlab-org/gitlab/-/issues/480905)

Request ID: 01JD1PHJ0WK1B7SKH630N2ZVY1 pmaports!5824 (merged) 19/11/2024 7:52 UTC +/- 2min

Request ID: 01JD2BTP1M5XJWXT9RC88KA7C4 pmaports!5823 (merged) 19/11/2024 14:05 +/- 1min

I've asked OSUOSL to restart the gitlab instance again and to set up a cronjob for daily restarts until it is resolved properly... (at 8:00 UTC, I think that way it won't affect most users). Not a great workaround, but better than having to ask them every other day to restart gitalb.

it was restarted, and we have the cronjob now

Side effect of the cronjob (that I did not realize at the time):

bpo jobs for building images (or packages) will fail if they happen during the restart:

[#1372394] 2024/11/21 08:08:11 Cloning repositories
Cloning into 'pmaports'...
remote: GitLab is not responding
fatal: unable to access 'https://gitlab.postmarketos.org/postmarketOS/pmaports.git/': The requested URL returned error: 502

(502 is the error code it gives when restarting)

So we need to manually restart these jobs if that happens. But still better than getting 500 errors on the site all the time. Let's hope this moves forward upstream and we can remove the cronjob soon.

restart cronjob sometimes caused gitlab to get stuck while rebooting (https://status.postmarketos.org/issues/2024-11-28-gitlab-booting/, https://status.postmarketos.org/issues/2024-12-02-gitlab-booting/)
OSUOSL disabled the cronjob again: "I've disabled that cronjob for now since it's hurting more than helping right now. I'll see if we can get more movement upstream on resolving this"

upstream issue is moving,

looks like a fix was ~~merged~~ created as MR, not merged yet and will be in the next gitlab release probably
they mentioned a potential workaround by setting an env var, which we'll try

We have the workaround env var mentioned in https://gitlab.com/gitlab-org/gitlab/-/issues/480905#note_2238870235 set now.

This has been deployed. Sorry it took longer to gitlab to restart. It should be back online now.

Let's hope this works!

Closing this issue

closed

I'm seeing 500 errors again today, reopening.

Not sure if they have the same cause or something different.

Good news is that the upstream fix was merged and should be in the next version.

reopened

we now have more CPU and RAM, which should help with solving this: https://status.postmarketos.org/issues/2024-12-08-gitlab-reboot-to-increase-ram-and-cpu/

I haven't seen a 500 error since, it is running much more stable now.

closed

500 errors

Child items 0

Activity