Caiguda de la base de dades del servidor

Hola!

Aquesta nit el servidor ha caigut a les 3 AM (horari BCN) i l’he tornat a aixecar. El que havia caigut era la base de dades amb aquest error:

/var/log/postgresql/postgresql-14-main.log

2024-11-19 02:21:11.268 UTC [1129105] akkoma@akkoma ERROR:  duplicate key value violates unique constraint "users_nickname_index"
2024-11-19 02:21:11.268 UTC [1129105] akkoma@akkoma DETAIL:  Key (nickname)=(privacy@lemmy.ml) already exists.
2024-11-19 02:21:11.268 UTC [1129105] akkoma@akkoma STATEMENT:  UPDATE "users" SET "name" = $1, "background" = $2, "nickname" = $3, "pinned_objects" = $4, "bio" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "updated_at" = $8 WHERE "id" = $9
2024-11-19 02:21:13.286 UTC [1128021] akkoma@akkoma ERROR:  duplicate key value violates unique constraint "users_nickname_index"
2024-11-19 02:21:13.286 UTC [1128021] akkoma@akkoma DETAIL:  Key (nickname)=(privacy@lemmy.ml) already exists.
2024-11-19 02:21:13.286 UTC [1128021] akkoma@akkoma STATEMENT:  UPDATE "users" SET "name" = $1, "background" = $2, "nickname" = $3, "pinned_objects" = $4, "bio" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "updated_at" = $8 WHERE "id" = $9
2024-11-19 02:21:14.312 UTC [1128021] akkoma@akkoma ERROR:  duplicate key value violates unique constraint "users_nickname_index"
2024-11-19 02:21:14.312 UTC [1128021] akkoma@akkoma DETAIL:  Key (nickname)=(privacy@lemmy.ml) already exists.
2024-11-19 02:21:14.312 UTC [1128021] akkoma@akkoma STATEMENT:  UPDATE "users" SET "name" = $1, "background" = $2, "nickname" = $3, "pinned_objects" = $4, "bio" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "updated_at" = $8 WHERE "id" = $9
2024-11-19 02:25:41.051 UTC [850] LOG:  server process (PID 1125619) was terminated by signal 9: Killed
2024-11-19 02:25:41.051 UTC [850] DETAIL:  Failed process was running: SELECT a0."id", a0."data", a0."local", a0."actor", a0."recipients", a0."inserted_at", a0."updated_at" FROM "activities" AS a0 WHERE (coalesce((a0."data")->'object'->>'id', (a0."data")->>'object') = $1) AND ((a0."data")->>'type' = $2)
2024-11-19 02:25:41.051 UTC [850] LOG:  terminating any other active server processes
2024-11-19 02:25:41.097 UTC [1130931] [unknown]@[unknown] LOG:  PID 1130904 in cancel request did not match any process
2024-11-19 02:25:41.098 UTC [1130932] akkoma@akkoma FATAL:  the database system is in recovery mode
2024-11-19 02:25:41.113 UTC [1130930] [unknown]@[unknown] LOG:  PID 1125619 in cancel request did not match any process
2024-11-19 02:25:41.149 UTC [1130933] akkoma@akkoma FATAL:  the database system is in recovery mode
2024-11-19 02:25:41.149 UTC [1130934] akkoma@akkoma FATAL:  the database system is in recovery mode
2024-11-19 02:25:41.190 UTC [850] LOG:  all server processes terminated; reinitializing
2024-11-19 02:25:41.325 UTC [850] LOG:  received fast shutdown request
2024-11-19 02:25:41.326 UTC [1130937] LOG:  database system was interrupted; last known up at 2024-11-19 02:22:15 UTC
2024-11-19 02:25:41.330 UTC [1130938] [unknown]@[unknown] LOG:  PID 1129882 in cancel request did not match any process
2024-11-19 02:25:41.348 UTC [1130939] akkoma@akkoma FATAL:  the database system is shutting down
2024-11-19 02:25:41.435 UTC [1130937] LOG:  database system was not properly shut down; automatic recovery in progress
2024-11-19 02:25:41.443 UTC [1130937] LOG:  redo starts at 23D/AAB1C4C0
2024-11-19 02:25:41.445 UTC [850] LOG:  abnormal database system shutdown
2024-11-19 02:25:41.451 UTC [850] LOG:  database system is shut down

Estic tenint una mica de dejà vu amb pleroma… :cry:

Estaré (estiguem) atent a noves caigudes…

cc @xaloc

3 'M'agrada'

Hmmm…, intentaré investigar. No se si serà allò de que hi ha prou processos per la base de dades o què.

1 'M'agrada'

Ui, aquest fil se m’havia escapat… Com ha acabat la cosa?

1 'M'agrada'

De moment no ha tornat a passar… Així està la cosa…

2 'M'agrada'

Ha tornat a passar, quan pugui torno a aixecar el servei.

EDIT: aixecat.

HI havia això al log de la base de dades:

2025-01-29 00:21:42.307 UTC [923277] akkoma@akkoma ERROR:  duplicate key value violates unique constraint "users_nickname_index"
2025-01-29 00:21:42.307 UTC [923277] akkoma@akkoma DETAIL:  Key (nickname)=(privacy@lemmy.ml) already exists.
2025-01-29 00:21:42.307 UTC [923277] akkoma@akkoma STATEMENT:  UPDATE "users" SET "name" = $1, "background" = $2, "nickname" = $3, "pinned_objects" = $4, "bio" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "updated_at" = $8 WHERE "id" = $9
2025-01-29 00:21:42.308 UTC [923269] akkoma@akkoma ERROR:  duplicate key value violates unique constraint "users_nickname_index"
2025-01-29 00:21:42.308 UTC [923269] akkoma@akkoma DETAIL:  Key (nickname)=(privacy@lemmy.ml) already exists.
2025-01-29 00:21:42.308 UTC [923269] akkoma@akkoma STATEMENT:  UPDATE "users" SET "name" = $1, "background" = $2, "nickname" = $3, "pinned_objects" = $4, "bio" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "updated_at" = $8 WHERE "id" = $9
2025-01-29 00:22:16.278 UTC [923277] akkoma@akkoma ERROR:  duplicate key value violates unique constraint "users_nickname_index"
2025-01-29 00:22:16.278 UTC [923277] akkoma@akkoma DETAIL:  Key (nickname)=(privacy@lemmy.ml) already exists.
2025-01-29 00:22:16.278 UTC [923277] akkoma@akkoma STATEMENT:  UPDATE "users" SET "name" = $1, "background" = $2, "nickname" = $3, "pinned_objects" = $4, "bio" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "updated_at" = $8 WHERE "id" = $9
2025-01-29 00:22:19.518 UTC [815] LOG:  server process (PID 922091) was terminated by signal 9: Killed
2025-01-29 00:22:19.518 UTC [815] DETAIL:  Failed process was running: SELECT o0."id", o0."data", o0."inserted_at", o0."updated_at" FROM "objects" AS o0 WHERE ((o0."data")->>'id' = $1)
2025-01-29 00:22:19.518 UTC [815] LOG:  terminating any other active server processes
2025-01-29 00:22:19.594 UTC [926915] [unknown]@[unknown] LOG:  PID 923731 in cancel request did not match any process
2025-01-29 00:22:19.596 UTC [926914] akkoma@akkoma FATAL:  the database system is in recovery mode
2025-01-29 00:22:19.612 UTC [926916] akkoma@akkoma FATAL:  the database system is in recovery mode
2025-01-29 00:22:19.648 UTC [926918] [unknown]@[unknown] LOG:  PID 924031 in cancel request did not match any process
2025-01-29 00:22:19.655 UTC [926920] akkoma@akkoma FATAL:  the database system is in recovery mode
2025-01-29 00:22:19.668 UTC [926919] akkoma@akkoma FATAL:  the database system is in recovery mode
2025-01-29 00:22:19.674 UTC [815] LOG:  all server processes terminated; reinitializing
2025-01-29 00:22:19.804 UTC [815] LOG:  received fast shutdown request
2025-01-29 00:22:19.805 UTC [926925] LOG:  database system was interrupted; last known up at 2025-01-29 00:19:02 UTC
2025-01-29 00:22:19.934 UTC [926925] LOG:  database system was not properly shut down; automatic recovery in progress
2025-01-29 00:22:19.943 UTC [926925] LOG:  redo starts at 2B6/A3811530
2025-01-29 00:22:19.945 UTC [815] LOG:  abnormal database system shutdown
2025-01-29 00:22:19.951 UTC [815] LOG:  database system is shut down

L’error de la duplicate key value se repito mucho hacia atrás. No sé si será eso.

3 'M'agrada'

Uep! Sembla que torna a estar caigut! @marcelcosta

Hola!

Aquest cop és degut a l’espai. Vaig a fer un upgrade:

Nov 03 19:34:18 bcn-fedi-cat systemd[1]: Starting PostgreSQL Cluster 14-main...
Nov 03 19:34:18 bcn-fedi-cat postgresql@14-main[1016317]: Error: /usr/lib/postgresql/14/bin/pg_ctl /usr/lib/postgresql/14/bin/pg_ctl start -D /var/lib/postgresql/14/main -l /var/log/postgresql/postgresql-14-ma>
Nov 03 19:34:18 bcn-fedi-cat postgresql@14-main[1016317]: 2025-11-03 19:34:18.307 UTC [1016322] FATAL:  could not write lock file "postmaster.pid": No space left on device
1 'M'agrada'

Li he donat el doble de capacitats (era la següent opció). També té el doble de cost.

1 'M'agrada'

Potser hauríem de fer números… Quan vam arrencar, teníem un full de càlcul “KiPagaKè” i potser fóra bo refer-lo.

2 'M'agrada'

i parlant de cost, estant en un altre continent, com puc contribuir als costos de la instància?

Vull dir, sense que es mengi la contribució les tarifes de transacció

1 'M'agrada'

Doncs per ara ho vehiculem a través de Fedicat - Open Collective

Aquest és un problema que comentàvem amb @ander_dapo i hem de veure com ho podem minimitzar.

Vaig fer una prova just ara mateix, de 10 euros. M’expliques quant van rebre?

Donçs, tinc missatges contradictoris, per una banda la targeta em diu que em van cobrar els 10 euros, i per altra banda open collective em diu que van declinar el cobrament. Però bé, per això serveixen les proves

Hola! Sí, a l’opencollective no apareix!

1 'M'agrada'