满盘后RabbitMQ恢复队列

如何解决满盘后RabbitMQ恢复队列

我们的一个 rabbitmq 持久队列变得太大并填满了整个磁盘 (>2 TB)。 RabbitMQ 崩溃了。腾出一些空间后，rabbitmq 重新启动，但未使用 vhost /。下面的日志似乎表明它找不到的进程有问题（日志中的 :noproc）。

这是一个生产服务器，一些队列包含重要数据。我怎样才能至少恢复一些队列？

CLI 日志：

$ rabbitmqctl restart_vhost --vhost /
Trying to restart vhost '/' on node 'rabbit@inoopa-storage' ...
Error:
Failed to start vhost '/' on node 'rabbit@inoopa-storage'Reason: {:shutdown,{:Failed_to_start_child,:rabbit_vhost_process,{:error,{{:noproc,{:gen_server2,:call,[#PID<10613.14642.53>,:out,:infinity]}},{:child,:undefined,:msg_store_persistent,{:rabbit_msg_store,:start_link,[:msg_store_persistent,'/home/rabbitmq/mnesia/rabbit@inoopa-storage/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L',[],{#Function<2.23124100/1 in :rabbit_queue_index>,{:start,[{:resource,"/",:queue,"workers_output_vector"},{:resource,"workers_scraped_keywords"},"workers_input_oxylabs"},"dev_ecommerce_detector_scrapped"},"ecommerce_detector_input"},"manual_input_facebook_not_priority"},"manual_input_facebook"},"manual_input_websitefinder"},"dev_ecommerce_detector_output"},"treatment_monitor"},"workers_filtered_keywords"},"dev2"},"elastic_worker_tmp"},"workers_input_scrapper"},"workers_input_oxylabs_problematic"},"best_phones_updater_input"},"best_sector_updater_input"},"workers_input_oxylabs_not_priority"},"dev3"},"workers_input_keywords"},"MoveQueue.py"},"manual_output_websitefinder"},...},{...},...]}}]},:transient,30000,:worker,[:rabbit_msg_store]}}}}}

消息存储显然已关闭：

$rabbitmqctl purge_queue dev
Purging queue 'dev' in vhost '/' ...
Error:
not_found

# From logs

2021-06-22 16:56:09.259 [info] <0.2025.0> Message store for directory '/home/rabbitmq/mnesia/rabbit@inoopa-storage/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped

重新启动虚拟主机时的完整服务器日志：

021-06-22 16:48:12.293 [info] <0.1471.0> Making sure data directory '/home/rabbitmq/mnesia/rabbit@inoopa-storage/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2021-06-22 16:48:12.298 [info] <0.1471.0> Starting message stores for vhost '/'
2021-06-22 16:48:12.298 [info] <0.1475.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2021-06-22 16:48:12.300 [info] <0.1471.0> Started message store of type transient for vhost '/'
2021-06-22 16:48:12.300 [info] <0.1478.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2021-06-22 16:48:15.320 [warning] <0.1478.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2021-06-22 16:48:15.700 [error] <0.885.0> ** Generic server <0.885.0> terminating
** Last message in was {'$gen_cast',{submit_async,#Fun<rabbit_queue_index.40.23124100>}}
** When Server state == undefined
** Reason for termination == 
** {function_clause,[{rabbit_queue_index,parse_segment_entries,[<<"G">>,false,{{array,16384,undefined,{{{{{{{true,<<131,221,139,210,241,243,169,198,3,109,103,34,100,173,40>>,... 96,240,167,250,99,4,116,114,117,101>>},no_del,no_ack},{{true,<<255,95,42,151,38,17,155,140,20,36,143,228,131,83,236,30>>,104,6,13,98,97,115,105,101,8,...>>},...}},...}],...]}
2021-06-22 16:48:15.706 [error] <0.885.0> CRASH REPORT Process <0.885.0> with 8 neighbours exited with reason: no function clause matching rabbit_queue_index:parse_segment_entries(<<"G">>,...>>,...}) line 1133 in gen_server2:terminate/3 line 1183
2021-06-22 16:48:15.709 [error] <0.350.0> Supervisor worker_pool_sup had child 5 started with worker_pool_worker:start_link(worker_pool) at <0.885.0> exit with reason no function clause matching rabbit_queue_index:parse_segment_entries(<<"G">>,...}) line 1133 in context child_terminated
2021-06-22 16:48:15.713 [error] <0.350.0> Supervisor worker_pool_sup had child 1 started with worker_pool_worker:start_link(worker_pool) at <0.882.0> exit with reason no function clause matching rabbit_queue_index:parse_segment_entries(<<"G">>,...}) line 1133 in context child_terminated
2021-06-22 16:48:15.716 [error] <0.350.0> Supervisor worker_pool_sup had child 4 started with worker_pool_worker:start_link(worker_pool) at <0.881.0> exit with reason no function clause matching rabbit_queue_index:parse_segment_entries(<<"G">>,...}) line 1133 in context child_terminated
2021-06-22 16:48:15.718 [error] <0.350.0> Supervisor worker_pool_sup had child 6 started with worker_pool_worker:start_link(worker_pool) at <0.884.0> exit with reason no function clause matching rabbit_queue_index:parse_segment_entries(<<"G">>,...}) line 1133 in context child_terminated
2021-06-22 16:48:15.719 [error] <0.1471.0> Failed to start message store of type msg_store_persistent for vhost '/': {{{function_clause,<<131...40,111,1,47,120,110,108,24,...]},...}
2021-06-22 16:48:15.721 [error] <0.350.0> Supervisor worker_pool_sup had child 3 started with worker_pool_worker:start_link(worker_pool) at <0.883.0> exit with reason no function clause matching rabbit_queue_index:parse_segment_entries(<<"G">>,...}) line 1133 in context child_terminated
2021-06-22 16:48:15.751 [error] <0.1478.0> CRASH REPORT Process <0.1478.0> with 1 neighbours exited with reason: {{function_clause,112,...} in gen_server2:init_it/6 line 608
2021-06-22 16:48:15.752 [error] <0.1491.0> ** Generic server <0.1491.0> terminating
** Last message in was {'$gen_cast',#Fun<rabbit_queue_index.40.23124100>}}
** When Server state == undefined
** Reason for termination == 
** {noproc,[{erlang,link,[<0.1487.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-1-',2,[{file,"src/rabbit_queue_index.erl"},{line,709}]},{worker_pool_worker,handle_cast,"src/worker_pool_worker.erl"},122}]},{gen_server2,handle_msg,"src/gen_server2.erl"},1067}]},{proc_lib,init_p_do_apply,"proc_lib.erl"},249}]}]}
2021-06-22 16:48:15.752 [error] <0.1492.0> ** Generic server <0.1492.0> terminating
** Last message in was {'$gen_cast',249}]}]}
2021-06-22 16:48:15.753 [error] <0.1489.0> ** Generic server <0.1489.0> terminating
** Last message in was {'$gen_cast',249}]}]}
2021-06-22 16:48:15.753 [error] <0.1493.0> ** Generic server <0.1493.0> terminating
** Last message in was {'$gen_cast',249}]}]}
2021-06-22 16:48:15.753 [error] <0.1490.0> ** Generic server <0.1490.0> terminating
** Last message in was {'$gen_cast',249}]}]}
2021-06-22 16:48:15.772 [error] <0.350.0> Supervisor worker_pool_sup had child 7 started with worker_pool_worker:start_link(worker_pool) at <0.887.0> exit with reason no function clause matching rabbit_queue_index:parse_segment_entries(<<"G">>,...}) line 1133 in context child_terminated
2021-06-22 16:48:15.772 [error] <0.1491.0> CRASH REPORT Process <0.1491.0> with 0 neighbours exited with reason: no such process or port in call to erlang:link(<0.1487.0>) in gen_server2:terminate/3 line 1183
2021-06-22 16:48:15.773 [error] <0.1492.0> CRASH REPORT Process <0.1492.0> with 0 neighbours exited with reason: no such process or port in call to erlang:link(<0.1487.0>) in gen_server2:terminate/3 line 1183
2021-06-22 16:48:15.773 [error] <0.1489.0> CRASH REPORT Process <0.1489.0> with 0 neighbours exited with reason: no such process or port in call to erlang:link(<0.1487.0>) in gen_server2:terminate/3 line 1183
2021-06-22 16:48:15.773 [error] <0.1493.0> CRASH REPORT Process <0.1493.0> with 0 neighbours exited with reason: no such process or port in call to erlang:link(<0.1487.0>) in gen_server2:terminate/3 line 1183
2021-06-22 16:48:15.773 [error] <0.1490.0> CRASH REPORT Process <0.1490.0> with 0 neighbours exited with reason: no such process or port in call to erlang:link(<0.1487.0>) in gen_server2:terminate/3 line 1183
2021-06-22 16:48:15.773 [error] <0.1494.0> ** Generic server <0.1494.0> terminating
** Last message in was {'$gen_cast',249}]}]}
2021-06-22 16:48:15.773 [error] <0.1494.0> CRASH REPORT Process <0.1494.0> with 0 neighbours exited with reason: no such process or port in call to erlang:link(<0.1487.0>) in gen_server2:terminate/3 line 1183
2021-06-22 16:48:15.799 [error] <0.350.0> Supervisor worker_pool_sup had child 8 started with worker_pool_worker:start_link(worker_pool) at <0.886.0> exit with reason no function clause matching rabbit_queue_index:parse_segment_entries(<<"G">>,...}) line 1133 in context child_terminated
2021-06-22 16:48:15.799 [error] <0.1495.0> ** Generic server <0.1495.0> terminating
** Last message in was {'$gen_cast',249}]}]}
2021-06-22 16:48:15.799 [error] <0.1495.0> CRASH REPORT Process <0.1495.0> with 0 neighbours exited with reason: no such process or port in call to erlang:link(<0.1487.0>) in gen_server2:terminate/3 line 1183
2021-06-22 16:48:15.800 [error] <0.350.0> Supervisor worker_pool_sup had child 2 started with worker_pool_worker:start_link(worker_pool) at <0.888.0> exit with reason no function clause matching rabbit_queue_index:parse_segment_entries(<<"G">>,...}) line 1133 in context child_terminated
2021-06-22 16:48:15.800 [error] <0.350.0> Supervisor worker_pool_sup had child 4 started with worker_pool_worker:start_link(worker_pool) at <0.1491.0> exit with reason no such process or port in call to erlang:link(<0.1487.0>) in context child_terminated
2021-06-22 16:48:15.800 [error] <0.1496.0> ** Generic server <0.1496.0> terminating
** Last message in was {'$gen_cast',249}]}]}
2021-06-22 16:48:15.800 [error] <0.1496.0> CRASH REPORT Process <0.1496.0> with 0 neighbours exited with reason: no such process or port in call to erlang:link(<0.1487.0>) in gen_server2:terminate/3 line 1183
2021-06-22 16:48:15.800 [error] <0.350.0> Supervisor worker_pool_sup had child 6 started with worker_pool_worker:start_link(worker_pool) at <0.1492.0> exit with reason no such process or port in call to erlang:link(<0.1487.0>) in context child_terminated
2021-06-22 16:48:15.800 [error] <0.1497.0> ** Generic server <0.1497.0> terminating
** Last message in was {'$gen_cast',249}]}]}
2021-06-22 16:48:15.801 [error] <0.1497.0> CRASH REPORT Process <0.1497.0> with 0 neighbours exited with reason: no such process or port in call to erlang:link(<0.1487.0>) in gen_server2:terminate/3 line 1183
2021-06-22 16:48:15.801 [error] <0.350.0> Supervisor worker_pool_sup had child 5 started with worker_pool_worker:start_link(worker_pool) at <0.1489.0> exit with reason no such process or port in call to erlang:link(<0.1487.0>) in context child_terminated
2021-06-22 16:48:15.801 [error] <0.1498.0> ** Generic server <0.1498.0> terminating
** Last message in was {'$gen_cast',249}]}]}
2021-06-22 16:48:15.801 [error] <0.1498.0> CRASH REPORT Process <0.1498.0> with 0 neighbours exited with reason: no such process or port in call to erlang:link(<0.1487.0>) in gen_server2:terminate/3 line 1183
...
2021-06-22 16:48:15.811 [error] <0.1469.0> Supervisor {<0.1469.0>,rabbit_vhost_sup_wrapper} had child rabbit_vhost_process started with rabbit_vhost_process:start_link(<<"/">>) at undefined exit with reason {error,{{{function_clause,...}} in context start_error
2021-06-22 16:48:15.906 [info] <0.1475.0> Message store for directory '/home/rabbitmq/mnesia/rabbit@inoopa-storage/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped