So I repeated the Insert Benchmark for Postgres 16.3 with a cached workload and a small server which is the same setup as reported here. This time I only ran it for two configurations:
- Throughput is similar for all benchmark steps
- KB written to storage per insert (wkbpi) are reduced with full_page_writes=off for the l.i1 and l.i2 benchmark steps that are write heavy. This is expected.
- Using full_page_writes=off doesn't fix the throughput variance problem and from the response time histograms the root cause remains the same (delete statements).
- LWN reported on this recently and I look forward to widespread support for atomic writes across SSD vendors with Linux. Even more so when this doesn't require a proprietary file system or vendor API. With such support everyone can safely use full_page_writes=off and turn off the InnoDB doublewrite buffer.
- If you scroll down on the IPS charts here you reach more charts for DPS (deletes/s) and max delete response time, both over 1 second intervals. While the max insert response time chart doesn't show variance, the max delete response time chart shows much variance. From the chart there are ~16 spikes at regular intervals over 3237 seconds or one per ~200 seconds.
From logs that I have not shared for the full_page_writes=off result:
- the l.i0 step started at 11:07 and finished at 11:13
- the l.x step finished at 11:15
- the l.i1 step finished at 12:09
- one checkpoint started and completed when it was running, then a second checkpoint started but did not finish (see below)
- many vacuums were done during this step and the interval between the spikes on the delete graphs aligns with the vacuum jobs listed below
- the l.i2 step finished at 14:07
- vacuum was one for the test table followed by checkpoint, this finished at 14:09
- the read-write test steps were done in sequence: qr100, qp100, ..., qr1000, qp1000 and qp1000 finished at 20:10, each runs for 1 hour
The following is from the test with full_page_writes=OFF.
Output from
grep checkpoint logfileis here and
grep "checkpoint starting" logfile output is below:
2024-06-02 11:07:38.612 UTC [1340363] LOG: checkpoint starting: immediate force wait
2024-06-02 11:37:39.041 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 12:07:39.938 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 12:37:39.817 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 13:07:39.765 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 13:37:39.609 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 14:07:39.717 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 14:08:04.767 UTC [1340363] LOG: checkpoint starting: immediate force wait
2024-06-02 14:38:05.017 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 15:08:05.261 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 15:38:05.265 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 16:08:05.181 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 16:38:05.211 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 17:08:05.436 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 17:38:05.545 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 18:08:05.581 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 18:38:05.401 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 19:08:05.801 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 19:38:05.757 UTC [1340363] LOG: checkpoint starting: time
2024-06-02 20:08:05.629 UTC [1340363] LOG: checkpoint starting: time
The output from
grep vacuum logfile is below and the spikes above for the per-second delete rates appear to align with the vacuum frequency show below. The first vacuums for the pi1 table are fast and more frquent because
index scans: 0 is shown for them -- so they do less work. Eventually vacuums that have
index scans: 1 are done every ~4 minutes.
Note that I recently began using
index_cleanup ON as an option to the manual vacuums done after the l.i2 step finishes, otherwise indexes are cleaned given the current heuristics. Unfortunately, I don't yet know whether there is a way (or whether I need to do it) for the automatic vacuums that represent the majority of the vacuum jobs listed below.
2024-06-02 11:07:39.879 UTC [1340408] LOG: automatic aggressive vacuum of table "template1.pg_catalog.pg_attribute": index scans: 0
2024-06-02 11:07:39.888 UTC [1340408] LOG: automatic aggressive vacuum of table "template1.pg_catalog.pg_class": index scans: 0
2024-06-02 11:07:39.892 UTC [1340408] LOG: automatic aggressive vacuum of table "template1.pg_catalog.pg_description": index scans: 0
2024-06-02 11:07:39.896 UTC [1340408] LOG: automatic aggressive vacuum of table "template1.pg_catalog.pg_depend": index scans: 1
2024-06-02 11:07:46.574 UTC [1340428] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:07:51.646 UTC [1340430] LOG: automatic aggressive vacuum of table "template1.pg_catalog.pg_statistic": index scans: 1
2024-06-02 11:07:56.670 UTC [1340433] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:08:06.671 UTC [1340437] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:08:16.673 UTC [1340447] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:08:26.674 UTC [1340451] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:08:36.675 UTC [1340464] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:08:46.678 UTC [1340474] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:08:56.679 UTC [1340478] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:09:06.680 UTC [1340482] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:09:16.681 UTC [1340492] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:09:26.681 UTC [1340496] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:09:36.782 UTC [1340509] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:09:46.882 UTC [1340519] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:09:56.982 UTC [1340523] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:10:06.983 UTC [1340528] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:10:16.985 UTC [1340538] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:10:26.986 UTC [1340542] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:10:37.088 UTC [1340555] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:10:47.089 UTC [1340565] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:10:57.091 UTC [1340569] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:11:07.092 UTC [1340573] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:11:27.213 UTC [1340588] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:11:47.316 UTC [1340611] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:12:07.319 UTC [1340619] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:12:30.655 UTC [1340634] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:12:50.621 UTC [1340657] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:13:10.625 UTC [1340671] LOG: automatic vacuum of table "ib.public.pi1": index scans: 0
2024-06-02 11:17:00.374 UTC [1341771] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 11:20:37.635 UTC [1341979] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 11:24:08.272 UTC [1342191] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 11:27:32.919 UTC [1342394] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 11:31:36.540 UTC [1342628] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 11:35:01.656 UTC [1342836] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 11:38:42.656 UTC [1343055] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 11:42:38.499 UTC [1343281] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 11:45:52.954 UTC [1343488] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 11:49:16.466 UTC [1343691] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 11:52:49.834 UTC [1343896] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 11:56:09.758 UTC [1344102] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 11:59:40.492 UTC [1344308] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 12:02:56.674 UTC [1344514] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 12:06:14.134 UTC [1344715] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 12:09:27.171 UTC [1344905] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 12:39:11.117 UTC [1347226] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 13:07:49.413 UTC [1349013] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 13:36:56.651 UTC [1350908] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 14:07:32.120 UTC [1352802] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 17:09:33.324 UTC [1586393] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 18:21:56.439 UTC [1811927] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 19:03:48.109 UTC [1813892] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1
2024-06-02 19:45:44.782 UTC [1946850] LOG: automatic vacuum of table "ib.public.pi1": index scans: 1