Caiguda de la base de dades del servidor

Nodes BCN.fedi.cat
1

Hola!

Aquesta nit el servidor ha caigut a les 3 AM (horari BCN) i l’he tornat a aixecar. El que havia caigut era la base de dades amb aquest error:

/var/log/postgresql/postgresql-14-main.log

2024-11-19 02:21:11.268 UTC [1129105] akkoma@akkoma ERROR:  duplicate key value violates unique constraint "users_nickname_index"
2024-11-19 02:21:11.268 UTC [1129105] akkoma@akkoma DETAIL:  Key (nickname)=(privacy@lemmy.ml) already exists.
2024-11-19 02:21:11.268 UTC [1129105] akkoma@akkoma STATEMENT:  UPDATE "users" SET "name" = $1, "background" = $2, "nickname" = $3, "pinned_objects" = $4, "bio" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "updated_at" = $8 WHERE "id" = $9
2024-11-19 02:21:13.286 UTC [1128021] akkoma@akkoma ERROR:  duplicate key value violates unique constraint "users_nickname_index"
2024-11-19 02:21:13.286 UTC [1128021] akkoma@akkoma DETAIL:  Key (nickname)=(privacy@lemmy.ml) already exists.
2024-11-19 02:21:13.286 UTC [1128021] akkoma@akkoma STATEMENT:  UPDATE "users" SET "name" = $1, "background" = $2, "nickname" = $3, "pinned_objects" = $4, "bio" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "updated_at" = $8 WHERE "id" = $9
2024-11-19 02:21:14.312 UTC [1128021] akkoma@akkoma ERROR:  duplicate key value violates unique constraint "users_nickname_index"
2024-11-19 02:21:14.312 UTC [1128021] akkoma@akkoma DETAIL:  Key (nickname)=(privacy@lemmy.ml) already exists.
2024-11-19 02:21:14.312 UTC [1128021] akkoma@akkoma STATEMENT:  UPDATE "users" SET "name" = $1, "background" = $2, "nickname" = $3, "pinned_objects" = $4, "bio" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "updated_at" = $8 WHERE "id" = $9
2024-11-19 02:25:41.051 UTC [850] LOG:  server process (PID 1125619) was terminated by signal 9: Killed
2024-11-19 02:25:41.051 UTC [850] DETAIL:  Failed process was running: SELECT a0."id", a0."data", a0."local", a0."actor", a0."recipients", a0."inserted_at", a0."updated_at" FROM "activities" AS a0 WHERE (coalesce((a0."data")->'object'->>'id', (a0."data")->>'object') = $1) AND ((a0."data")->>'type' = $2)
2024-11-19 02:25:41.051 UTC [850] LOG:  terminating any other active server processes
2024-11-19 02:25:41.097 UTC [1130931] [unknown]@[unknown] LOG:  PID 1130904 in cancel request did not match any process
2024-11-19 02:25:41.098 UTC [1130932] akkoma@akkoma FATAL:  the database system is in recovery mode
2024-11-19 02:25:41.113 UTC [1130930] [unknown]@[unknown] LOG:  PID 1125619 in cancel request did not match any process
2024-11-19 02:25:41.149 UTC [1130933] akkoma@akkoma FATAL:  the database system is in recovery mode
2024-11-19 02:25:41.149 UTC [1130934] akkoma@akkoma FATAL:  the database system is in recovery mode
2024-11-19 02:25:41.190 UTC [850] LOG:  all server processes terminated; reinitializing
2024-11-19 02:25:41.325 UTC [850] LOG:  received fast shutdown request
2024-11-19 02:25:41.326 UTC [1130937] LOG:  database system was interrupted; last known up at 2024-11-19 02:22:15 UTC
2024-11-19 02:25:41.330 UTC [1130938] [unknown]@[unknown] LOG:  PID 1129882 in cancel request did not match any process
2024-11-19 02:25:41.348 UTC [1130939] akkoma@akkoma FATAL:  the database system is shutting down
2024-11-19 02:25:41.435 UTC [1130937] LOG:  database system was not properly shut down; automatic recovery in progress
2024-11-19 02:25:41.443 UTC [1130937] LOG:  redo starts at 23D/AAB1C4C0
2024-11-19 02:25:41.445 UTC [850] LOG:  abnormal database system shutdown
2024-11-19 02:25:41.451 UTC [850] LOG:  database system is shut down

Estic tenint una mica de dejà vu amb pleroma… :cry:

Estaré (estiguem) atent a noves caigudes…

cc @xaloc

3 'M'agrada'
2

Hmmm…, intentaré investigar. No se si serà allò de que hi ha prou processos per la base de dades o què.

1 'M'agrada'
3

Ui, aquest fil se m’havia escapat… Com ha acabat la cosa?

1 'M'agrada'
4

De moment no ha tornat a passar… Així està la cosa…

2 'M'agrada'
5

Ha tornat a passar, quan pugui torno a aixecar el servei.

EDIT: aixecat.

HI havia això al log de la base de dades:

2025-01-29 00:21:42.307 UTC [923277] akkoma@akkoma ERROR:  duplicate key value violates unique constraint "users_nickname_index"
2025-01-29 00:21:42.307 UTC [923277] akkoma@akkoma DETAIL:  Key (nickname)=(privacy@lemmy.ml) already exists.
2025-01-29 00:21:42.307 UTC [923277] akkoma@akkoma STATEMENT:  UPDATE "users" SET "name" = $1, "background" = $2, "nickname" = $3, "pinned_objects" = $4, "bio" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "updated_at" = $8 WHERE "id" = $9
2025-01-29 00:21:42.308 UTC [923269] akkoma@akkoma ERROR:  duplicate key value violates unique constraint "users_nickname_index"
2025-01-29 00:21:42.308 UTC [923269] akkoma@akkoma DETAIL:  Key (nickname)=(privacy@lemmy.ml) already exists.
2025-01-29 00:21:42.308 UTC [923269] akkoma@akkoma STATEMENT:  UPDATE "users" SET "name" = $1, "background" = $2, "nickname" = $3, "pinned_objects" = $4, "bio" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "updated_at" = $8 WHERE "id" = $9
2025-01-29 00:22:16.278 UTC [923277] akkoma@akkoma ERROR:  duplicate key value violates unique constraint "users_nickname_index"
2025-01-29 00:22:16.278 UTC [923277] akkoma@akkoma DETAIL:  Key (nickname)=(privacy@lemmy.ml) already exists.
2025-01-29 00:22:16.278 UTC [923277] akkoma@akkoma STATEMENT:  UPDATE "users" SET "name" = $1, "background" = $2, "nickname" = $3, "pinned_objects" = $4, "bio" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "updated_at" = $8 WHERE "id" = $9
2025-01-29 00:22:19.518 UTC [815] LOG:  server process (PID 922091) was terminated by signal 9: Killed
2025-01-29 00:22:19.518 UTC [815] DETAIL:  Failed process was running: SELECT o0."id", o0."data", o0."inserted_at", o0."updated_at" FROM "objects" AS o0 WHERE ((o0."data")->>'id' = $1)
2025-01-29 00:22:19.518 UTC [815] LOG:  terminating any other active server processes
2025-01-29 00:22:19.594 UTC [926915] [unknown]@[unknown] LOG:  PID 923731 in cancel request did not match any process
2025-01-29 00:22:19.596 UTC [926914] akkoma@akkoma FATAL:  the database system is in recovery mode
2025-01-29 00:22:19.612 UTC [926916] akkoma@akkoma FATAL:  the database system is in recovery mode
2025-01-29 00:22:19.648 UTC [926918] [unknown]@[unknown] LOG:  PID 924031 in cancel request did not match any process
2025-01-29 00:22:19.655 UTC [926920] akkoma@akkoma FATAL:  the database system is in recovery mode
2025-01-29 00:22:19.668 UTC [926919] akkoma@akkoma FATAL:  the database system is in recovery mode
2025-01-29 00:22:19.674 UTC [815] LOG:  all server processes terminated; reinitializing
2025-01-29 00:22:19.804 UTC [815] LOG:  received fast shutdown request
2025-01-29 00:22:19.805 UTC [926925] LOG:  database system was interrupted; last known up at 2025-01-29 00:19:02 UTC
2025-01-29 00:22:19.934 UTC [926925] LOG:  database system was not properly shut down; automatic recovery in progress
2025-01-29 00:22:19.943 UTC [926925] LOG:  redo starts at 2B6/A3811530
2025-01-29 00:22:19.945 UTC [815] LOG:  abnormal database system shutdown
2025-01-29 00:22:19.951 UTC [815] LOG:  database system is shut down

L’error de la duplicate key value se repito mucho hacia atrás. No sé si será eso.

3 'M'agrada'