Problem: After database migration
SELECT * FROM <table_name>
fails with the following message:
ERROR: missing chunk number 0 for toast value 123456 in pg_toast_2619
At first, I found https://gist.github.com/supix/80f9a6111dc954cf38ee99b9dedf187a and https://newbiedba.wordpress.com/2015/07/07/postgresql-missing-chunk-0-for-toast-value-in-pg_toast/, but I could not isolate corrupted row, i.e. every query of the following form
SELECT * FROM <table_name> ORDER BY id LIMIT 5000 OFFSET 0; SELECT * FROM <table_name> ORDER BY id LIMIT 5000 offset 5000; ...even
SELECT * FROM <table_name> ORDER BY id LIMIT 1;failed with the same error message.
A second article (see https://fluca1978.github.io/2021/02/08/PostgreSQLToastCorruption) together with a set of Postgres functions for detecting corrupted toasts (see https://gitlab.com/fluca1978/fluca1978-pg-utils/-/blob/master/examples/toast/find_bad_toast.sql) inspired me how to find the table backed by `pg_toast_2619`, since it was already clear that the problem was not in my <table_name>:
SELECT reltoastrelid, reltoastrelid::regclass, pg_relation_filepath( reltoastrelid::regclass ), relname
FROM pg_class
WHERE reltoastrelid = 'pg_toast.pg_toast_2619'::regclass
AND relkind = 'r';
The result was pg_statistic and I executed the inverse query to recheck:
SELECT reltoastrelid, reltoastrelid::regclass, pg_relation_filepath( reltoastrelid::regclass ), relname
FROM pg_class
WHERE relname = 'pg_statistic'
AND relkind = 'r';
the result was pg_toast_2619. Hence the problem was not in <table_name> but in pg_statistic, more precisely in the TOAST of pg_statistic. But why the SELECT * FROM <tabel_name> generated this problem? Because Postgres uses pg_statistic to plan the query. Meanwhile a collegue of mine
found https://www.ongres.com/blog/solving-postgresql-statistic-corruption/. This article shows how to deal with such cases but also shows that pg_toast_2619 is somehow constant between different Postgres installations.
What I learned from this problem:
- How to find a table by its corresponding TOAST.
- There are hidden actions behind a simple SELECT * FROM <tabel_name>, i.e. PostgreSQL can perform a reading from another table first and then from the table `<tabel_name>` (not everything is as it seems at first sight).