Skip to content

Rodrigo Urubatan – About Code

Helping ruby developers to use the best tools for each job so they can solve hard problems, with less bugs and have more free time.

Menu
  • Home
  • My last presentations
  • About
  • Privacy Policy
Menu

I see Dead Jobs everywhere (sidekiq DeadSet)

Posted on 2023-02-072023-02-07 by Rodrigo Urubatan

When you are using Sidekiq to handle asynchronous jobs, some times there are exceptions and jobs failing, and I say sometimes because your environment is probably perfect, there is no lag, all services your jobs depend on are always on and responsive, and you probably write better code than most other developers ๐Ÿ˜› otherwise it probably happens quite often…

But Sidekiq will retry that job for you, of course this is configurable:

# Retry 5 times before giving up
sidekiq_options retry: 5
# Don't retry at all
sidekiq_options retry: false

The default number of retries is 25, and with that many tries most of the transient problems will fix itselves probably.

But sometimes not even being that insistent is enough to avoid the problem, and in that case sidekiq will send your job to a DeadSet where all the job ghosts live.

For most applications, it is good to know when some or all jobs go to that DeadSet, meaning Sidekiq will not retry it anymore.

In this situation you can log the error, send one email for a human to fix the issue by hand, or handle it the best way possible for your business.

To intercept this, sidekiq the sidekiq_retries_exhausted hook, that you can configure either per worker class as bellow:

class ImportantWorker
  include Sidekiq::Worker
  sidekiq_options retry: 5
 
  sidekiq_retries_exhausted do |msg, exception|
    # example with using Rails' logger
    Rails.logger.warn("Failed #{msg['class']} with #{msg['args']}: #{msg['error_message']}", error: exception)
  end
 
  def perform(important_arguments)
    # do some work
  end
end

or you can configure a global handler for the entire application, adding a handler to Sidekiq like bellow:

Sidekiq.configure_server do |config|
  # other config stuff...
 
  config.death_handlers << ->(job, ex) do
    Rails.logger.error "Surprise, an error!, #{job['class']} #{job["jid"]} just died with error #{ex.message}."
  end
end

I usually add this code to the config/initializers/sidekiq.rb file where all the sidekiq related configurations live together.

Of course just logging like this will not solve your problem, but intercepting the dead jobs event will allow you to write more robust asynchronous job processing applications.

Please add a comment bellow if you have any other issues with asynchronous jobs that I could help with.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recent posts

  • I see Dead Jobs everywhere (sidekiq DeadSet)
  • Quick tips that help: rails notes
  • Ruby 3.2.0 released with WASI, YJIT, RegExp improvements, …
  • Rubyconf Thailand quick summary and slides
  • Testing download contents with Cucumber+Capybara and chromedriver

Comments

  1. When Kubernetes is not the right choice? | Rodrigo Urubatan - About Code on Rails from “zero” to kubernetes – first pod
  2. When Kubernetes is not the right choice? | Rodrigo Urubatan - About Code on How to use docker to have an uniform development environment for your rails project
  3. Rails from "zero" to kubernetes โ€“ ingress reverse proxy | Rodrigo Urubatan - About Code on Rails from “zero” to kubernetes – a service to access your pod
  4. Rails from "zero" to kubernetes โ€“ horizontal autoscaling | Rodrigo Urubatan - About Code on Rails from “zero” to kubernetes – a service to access your pod
  5. Jeronimo on 6 Lessons From CrossFit That will help your developer career (Or any other career in the matter of fact)

Arquives

  • February 2023
  • January 2023
  • December 2022
  • June 2021
  • March 2020
  • January 2020
  • July 2019
  • June 2019
  • May 2019
  • October 2018
  • September 2018
  • August 2018
  • July 2018
  • June 2018
  • May 2018
  • February 2018
  • January 2018
  • November 2017
  • August 2015
  • August 2014
  • July 2014
  • August 2007

Categories

  • articles
  • cfp
  • firebase
  • gems
  • git
  • opinion
  • presentations
  • projects
  • rails6
  • ruby
  • Sem categoria
  • server-api
  • Uncategorized
© 2023 Rodrigo Urubatan – About Code | Powered by Minimalist Blog WordPress Theme