Hello MQ

外观

重试与 DLQ ​

本页结论：重试与死信队列（DLQ）在四个产品里是三种不同机制，不可互相类比：RabbitMQ 没有内置消费重试，靠 TTL + DLX 组合出重试环；RocketMQ 由 Broker 内置按档位重试并自动转死信；Kafka 完全没有内置重试，重试与 DLQ 都是应用层用 retry/DLQ topic 搭出来的。DLQ 是隔离区与证据柜，不是垃圾桶——进 DLQ 之后仍需告警、人工定责与回放。

为什么需要「有限重试 + 隔离」 ​

消费失败分两类：

  • 瞬时故障（下游超时、锁冲突、网络抖动）：重试大概率成功；
  • 确定性失败（Schema 缺字段、引用数据不存在、代码必错路径，即毒消息）：重试一万次也不会成功。

对毒消息无限重试/requeue 会卡住队列头部，阻塞后面所有正常消息。正确形态是：有限次数重试（带延迟），超过上限后把消息移入 DLQ 隔离，保留原始内容供人工处理。本仓库毒消息实验完整复现了这条链路：

rabbitmq / retry-dlqbash demos/rabbitmq/retry-dlq/run.sh
assert.out.txt
PASS confirmed: 3
PASS business_rows: 2
PASS poisonAttempts: 1,2,3
PASS poisonMovedToDlq: 1
PASS dlqMessages: 1
PASS workQueueDepthAfter: 0
PASS retryQueueDepthAfter: 0
PASS consumer.exitCode: 0
RESULT: all assertions passed
consumer.out.txt
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[consumer] timestamp=2026-10-07T10:28:15.088784369Z level=INFO service=order-service product=rabbitmq lab=retry-dlq messageId=01JMQ000000000000000000001 eventType=order.created schemaVersion=1 aggregateId=order-1001 traceId=00000000000000000000000000000001 correlationId=order-1001 destination=orders.work consumer=consumer-1 attempt=1 redelivered=false status=received
[replay-gate] timestamp=2026-10-07T10:28:15.101327476Z checkpoint=after-delivery messageId=01JMQ000000000000000000001 token=fb78909d7c14-1-after-delivery-01JMQ000000000000000000001-1 status=waiting
[replay-gate] timestamp=2026-10-07T10:28:16.408009254Z checkpoint=after-delivery messageId=01JMQ000000000000000000001 token=fb78909d7c14-1-after-delivery-01JMQ000000000000000000001-1 status=released
[consumer] timestamp=2026-10-07T10:28:16.419738210Z level=INFO service=order-service product=rabbitmq lab=retry-dlq messageId=01JMQ000000000000000000001 eventType=order.created schemaVersion=1 aggregateId=order-1001 traceId=00000000000000000000000000000001 correlationId=order-1001 destination=orders.work attempt=1 status=business_committed
[replay-gate] timestamp=2026-10-07T10:28:16.420247190Z checkpoint=before-ack messageId=01JMQ000000000000000000001 token=fb78909d7c14-1-before-ack-01JMQ000000000000000000001-2 status=waiting
[replay-gate] timestamp=2026-10-07T10:28:17.722595981Z checkpoint=before-ack messageId=01JMQ000000000000000000001 token=fb78909d7c14-1-before-ack-01JMQ000000000000000000001-2 status=released
[consumer] timestamp=2026-10-07T10:28:17.724756019Z level=INFO service=order-service product=rabbitmq lab=retry-dlq messageId=01JMQ000000000000000000002 eventType=order.created schemaVersion=1 aggregateId=order-1002 traceId=00000000000000000000000000000002 correlationId=order-1002 destination=orders.work consumer=consumer-1 attempt=1 redelivered=false status=received
[replay-gate] timestamp=2026-10-07T10:28:17.725346081Z checkpoint=after-delivery messageId=01JMQ000000000000000000002 token=fb78909d7c14-1-after-delivery-01JMQ000000000000000000002-3 status=waiting
[replay-gate] timestamp=2026-10-07T10:28:19.130247171Z checkpoint=after-delivery messageId=01JMQ000000000000000000002 token=fb78909d7c14-1-after-delivery-01JMQ000000000000000000002-3 status=released
[consumer] timestamp=2026-10-07T10:28:19.132336747Z level=INFO service=order-service product=rabbitmq lab=retry-dlq messageId=01JMQ000000000000000000002 eventType=order.created schemaVersion=1 aggregateId=order-1002 traceId=00000000000000000000000000000002 correlationId=order-1002 destination=orders.work attempt=1 status=business_committed
[replay-gate] timestamp=2026-10-07T10:28:19.132789332Z checkpoint=before-ack messageId=01JMQ000000000000000000002 token=fb78909d7c14-1-before-ack-01JMQ000000000000000000002-4 status=waiting
[replay-gate] timestamp=2026-10-07T10:28:20.636430076Z checkpoint=before-ack messageId=01JMQ000000000000000000002 token=fb78909d7c14-1-before-ack-01JMQ000000000000000000002-4 status=released
[consumer] timestamp=2026-10-07T10:28:20.637943339Z level=INFO service=order-service product=rabbitmq lab=retry-dlq messageId=01JMQ00000000000000000000P eventType=order.created schemaVersion=1 aggregateId=order-poison traceId=0000000000000000000000000000000P correlationId=order-poison destination=orders.work consumer=consumer-1 attempt=1 redelivered=false status=received
[replay-gate] timestamp=2026-10-07T10:28:20.638491974Z checkpoint=after-delivery messageId=01JMQ00000000000000000000P token=fb78909d7c14-1-after-delivery-01JMQ00000000000000000000P-5 status=waiting
[replay-gate] timestamp=2026-10-07T10:28:22.040822172Z checkpoint=after-delivery messageId=01JMQ00000000000000000000P token=fb78909d7c14-1-after-delivery-01JMQ00000000000000000000P-5 status=released
[consumer] timestamp=2026-10-07T10:28:22.041904172Z level=INFO service=order-service product=rabbitmq lab=retry-dlq messageId=01JMQ00000000000000000000P eventType=order.created schemaVersion=1 aggregateId=order-poison traceId=0000000000000000000000000000000P correlationId=order-poison destination=orders.work attempt=1 status=retry
[replay-gate] timestamp=2026-10-07T10:28:22.042433650Z checkpoint=before-retry messageId=01JMQ00000000000000000000P token=fb78909d7c14-1-before-retry-01JMQ00000000000000000000P-6 status=waiting
[replay-gate] timestamp=2026-10-07T10:28:23.344547194Z checkpoint=before-retry messageId=01JMQ00000000000000000000P token=fb78909d7c14-1-before-retry-01JMQ00000000000000000000P-6 status=released
[consumer] timestamp=2026-10-07T10:28:24.347460284Z level=INFO service=order-service product=rabbitmq lab=retry-dlq messageId=01JMQ00000000000000000000P eventType=order.created schemaVersion=1 aggregateId=order-poison traceId=0000000000000000000000000000000P correlationId=order-poison destination=orders.work consumer=consumer-1 attempt=2 redelivered=false status=received
[replay-gate] timestamp=2026-10-07T10:28:24.348008177Z checkpoint=after-delivery messageId=01JMQ00000000000000000000P token=fb78909d7c14-1-after-delivery-01JMQ00000000000000000000P-7 status=waiting
[replay-gate] timestamp=2026-10-07T10:28:25.651611504Z checkpoint=after-delivery messageId=01JMQ00000000000000000000P token=fb78909d7c14-1-after-delivery-01JMQ00000000000000000000P-7 status=released
[consumer] timestamp=2026-10-07T10:28:25.652492668Z level=INFO service=order-service product=rabbitmq lab=retry-dlq messageId=01JMQ00000000000000000000P eventType=order.created schemaVersion=1 aggregateId=order-poison traceId=0000000000000000000000000000000P correlationId=order-poison destination=orders.work attempt=2 status=retry
[replay-gate] timestamp=2026-10-07T10:28:25.652931477Z checkpoint=before-retry messageId=01JMQ00000000000000000000P token=fb78909d7c14-1-before-retry-01JMQ00000000000000000000P-8 status=waiting
[replay-gate] timestamp=2026-10-07T10:28:27.355753599Z checkpoint=before-retry messageId=01JMQ00000000000000000000P token=fb78909d7c14-1-before-retry-01JMQ00000000000000000000P-8 status=released
[consumer] timestamp=2026-10-07T10:28:28.358190936Z level=INFO service=order-service product=rabbitmq lab=retry-dlq messageId=01JMQ00000000000000000000P eventType=order.created schemaVersion=1 aggregateId=order-poison traceId=0000000000000000000000000000000P correlationId=order-poison destination=orders.work consumer=consumer-1 attempt=3 redelivered=false status=received
[replay-gate] timestamp=2026-10-07T10:28:28.358640906Z checkpoint=after-delivery messageId=01JMQ00000000000000000000P token=fb78909d7c14-1-after-delivery-01JMQ00000000000000000000P-9 status=waiting
[replay-gate] timestamp=2026-10-07T10:28:29.661027280Z checkpoint=after-delivery messageId=01JMQ00000000000000000000P token=fb78909d7c14-1-after-delivery-01JMQ00000000000000000000P-9 status=released
[replay-gate] timestamp=2026-10-07T10:28:29.662056362Z checkpoint=before-dlq messageId=01JMQ00000000000000000000P token=fb78909d7c14-1-before-dlq-01JMQ00000000000000000000P-10 status=waiting
[replay-gate] timestamp=2026-10-07T10:28:31.064384155Z checkpoint=before-dlq messageId=01JMQ00000000000000000000P token=fb78909d7c14-1-before-dlq-01JMQ00000000000000000000P-10 status=released
[consumer] timestamp=2026-10-07T10:28:31.066674421Z level=INFO service=order-service product=rabbitmq lab=retry-dlq messageId=01JMQ00000000000000000000P eventType=order.created schemaVersion=1 aggregateId=order-poison traceId=0000000000000000000000000000000P correlationId=order-poison destination=orders.work attempt=3 status=poison_to_dlq
[consumer] timestamp=2026-10-07T10:28:31.067017057Z level=INFO service=order-service product=rabbitmq lab=retry-dlq queue=orders.work received=5 status=done
inspect-db.out.txt
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[inspect] timestamp=2026-10-07T10:28:31.977614051Z level=INFO service=order-service product=rabbitmq lab=retry-dlq business_rows=2 processed_rows=2 status=snapshot
producer.out.txt
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[producer] timestamp=2026-10-07T10:28:14.239167174Z level=INFO service=order-service product=rabbitmq lab=retry-dlq messageId=01JMQ000000000000000000001 eventType=order.created schemaVersion=1 aggregateId=order-1001 traceId=00000000000000000000000000000001 correlationId=order-1001 destination=orders.work routingKey=orders.work durationMs=7 status=confirmed
[producer] timestamp=2026-10-07T10:28:14.243278287Z level=INFO service=order-service product=rabbitmq lab=retry-dlq messageId=01JMQ000000000000000000002 eventType=order.created schemaVersion=1 aggregateId=order-1002 traceId=00000000000000000000000000000002 correlationId=order-1002 destination=orders.work routingKey=orders.work durationMs=0 status=confirmed
[producer] timestamp=2026-10-07T10:28:14.244510557Z level=INFO service=order-service product=rabbitmq lab=retry-dlq messageId=01JMQ00000000000000000000P eventType=order.created schemaVersion=1 aggregateId=order-poison traceId=0000000000000000000000000000000P correlationId=order-poison destination=orders.work routingKey=orders.work durationMs=0 status=confirmed
[producer] timestamp=2026-10-07T10:28:14.244652672Z level=INFO service=order-service product=rabbitmq lab=retry-dlq destination=orders.work confirmed=3 status=done

实验拓扑与断言细节见 RabbitMQ 可靠性。

三机制对比（RabbitMQ / RocketMQ / Kafka） ​

维度RabbitMQ：TTL + DLX 组合RocketMQ：Broker 内置Kafka：应用层
机制work 队列 nack(requeue=false) → DLX 转入 retry 队列（TTL）→ 到期 dead-letter 回 workBroker 自动把失败消息投入 %RETRY%<group> 主题，按档位延迟后再投递消费者自己捕获异常，发到 retry topic（可多级），超限发 DLQ topic
延迟控制retry 队列的 x-message-ttl（队列级，到期从队头计算）Broker 重试档位（递增延迟）应用自定义：多级 retry topic + 延迟手段（外部调度/时间轮）
次数记录消息头 x-death（Broker 每次 dead-letter 追加）+ 应用计数Broker 侧记录重投次数消息头里由应用自行携带 attempt
DLQ 形成应用判断 attempt ≥ 上限后显式发布到 DLQ 队列并 ACK 原消息超过最大重试次数 Broker 自动转入 %DLQ%<group>应用显式 produce 到 DLQ topic
对顺序的影响dead-letter 会改变消息在队列中的位置，重试环中无严格队头顺序承诺重试消息脱离原消费顺序retry/DLQ topic 与原分区无顺序关系

三个机制不是同一功能的不同配置，写文档与选型时不得互相替换描述。

重试预算与退避 ​

  • 区分故障类型：下游超时按瞬时故障重试；Schema 校验失败这类确定性错误应尽早转 DLQ，不要浪费重试预算（毒消息实验里 attempt 1 即可判定）。
  • 带延迟的重试：立即重试往往打在同一个故障点上；指数退避或固定档位能显著降低重投风暴风险。
  • 上限必须显式：max-attempts 是系统参数而不是代码品味；改上限要同步评估 DLQ 增量与告警阈值。
  • 重试也要幂等：每次重试都可能撞上「业务已提交、确认未发出」窗口，重试路径必须复用幂等消费流程。

DLQ 治理 ​

DLQ 只完成「隔离」，治理闭环在 DLQ 之外：

  1. 告警：DLQ 深度、新增速率、最老消息年龄都要有阈值告警（指标定义见可观测性）。
  2. 留证：进 DLQ 时同步记录失败原因、attempt 历史、原始 traceId——DLQ 里只有消息本体，原因在日志里。
  3. 回放：修复问题后从 DLQ 重放是常规操作；回放等价于重新投递，接收端幂等仍然必须成立。
  4. 保留期：DLQ 消息应有保留策略，超期归档到冷存储，而不是静默删除。

保证成立的条件 / 不保证什么 ​

  • 条件：重试次数可观测（x-death / Broker 计数 / 应用头字段）；DLQ 有告警与回放流程；重试路径幂等。
  • 不保证：重试后消息仍保持原队列顺序；进入 DLQ 的消息会被自动处理；RabbitMQ 队列级 TTL 能表达任意精细的逐条延迟（到期从队头计算，见 RabbitMQ 可靠性）。

常见误区 ​

  • 「nack(requeue=true) 一直重试就行」——无延迟、无计数，毒消息无限占用队头。
  • 「四个产品都有内置消费重试」——只有 RocketMQ 是 Broker 内置；RabbitMQ 是队列参数组合，Kafka 完全在应用层。
  • 「消息进了 DLQ 就没事了」——DLQ 不处理消息，只是把问题从「阻塞」变成「待办」；没有告警的 DLQ 等于黑洞。
  • 「DLQ 里的消息可以直接删」——删除前先确认业务侧确实不需要补偿或审计。

官方资料 ​

以统一实验验证消息系统语义边界

MIT License