Skip to content

Job Daemon Configuration

The Job Daemon is responsible for executing jobs in Clarive. It can be configured with various options to control its behavior.

Configuration Options

frequency

  • Label: Job Server Frequency
  • Type: Integer
  • Default: 5 seconds
  • Description: The frequency in seconds at which the job daemon checks for new jobs to execute.

iterations

  • Label: Job Server Iterations before restart
  • Type: Integer
  • Default: 1000 iterations
  • Description: The number of iterations the job daemon will run before automatically restarting. This helps prevent memory leaks and keeps the daemon running smoothly.

wait_for_killed

  • Label: Seconds to wait before declaring job killed
  • Type: Integer
  • Default: 10 seconds
  • Description: The time in seconds the daemon waits before declaring a job as killed if it doesn't respond.

mode

  • Label: Job Spawn Mode (spawn,fork,detach)
  • Type: String
  • Default: 'fork'
  • Description: Determines how jobs are executed. Options are:
  • spawn: Runs jobs as separate background processes
  • fork: Forks a new process for each job
  • detach: Forks and detaches the process from the parent

unified_log

  • Label: Set true to have jobs report to dispatcher log
  • Type: Boolean
  • Default: False (0)
  • Description: When enabled, job logs are reported to the dispatcher log instead of individual job logs.

job_host_affinity

  • Label: All steps of jobs must been executed in the same host
  • Type: Boolean
  • Default: True (1)
  • Description: Ensures that all steps of a job are executed on the same host. This is important for jobs that require consistent environment state.

id

  • Label: All steps of jobs must been executed in the same host
  • Default: System hostname (lowercase)
  • Description: The identifier for this job daemon instance. Used to ensure job host affinity.

boost

  • Label: Number of jobs to be executed in the same daemon iteration
  • Default: 10 jobs
  • Description: The number of jobs that can be executed in a single daemon iteration cycle.

auto_throttle

  • Label: Automatically adjust pubsub throttle based on active jobs
  • Type: Boolean
  • Default: False (0)
  • Description: When enabled, automatically adjusts the pubsub throttle based on the number of active jobs to prevent system overload.

throttle_jobs

  • Label: Number of active jobs to rise the the pubsub throttle
  • Type: Integer
  • Default: 25 jobs
  • Description: The number of active jobs that triggers an increase in pubsub throttle when auto_throttle is enabled.

throttle_increment

  • Label: Number of miliseconds to rise when the throttle_jobs is reached
  • Type: Integer
  • Default: 333 milliseconds
  • Description: The amount of time in milliseconds to increase the pubsub throttle when throttle_jobs threshold is reached.

max_concurrent_jobs

  • Label: Number of concurrent jobs allowed by dispatcher (0 => unlimitted)
  • Type: Integer
  • Default: 0 (unlimited)
  • Description: The maximum number of jobs that can run concurrently. Set to 0 for unlimited concurrent jobs.

dynamic_job_slots

  • Label: Calculate the job slots available dynamically
  • Type: Boolean
  • Default: True (1)
  • Description: When enabled, calculates the available job slots dynamically based on system memory usage.

post_weight

  • Label: Weight assigned to POST jobs in daemon query
  • Default: 0
  • Description: The weight assigned to POST jobs when determining job execution priority. Higher values mean lower priority.

run_weight

  • Label: Weight assigned to RUN jobs in daemon query
  • Default: 0.1
  • Description: The weight assigned to RUN jobs when determining job execution priority. Higher values mean lower priority.

pre_weight

  • Label: Weight assigned to PRE jobs in daemon query
  • Default: 0.2
  • Description: The weight assigned to PRE jobs when determining job execution priority. Higher values mean lower priority.

How Job Priority Works

The job daemon uses a weighting system to determine which jobs to execute first. Jobs with lower weights are executed before jobs with higher weights. The weight is calculated based on the job's endtime and step type, multiplied by the configured weights for each step type.

Memory Management

When dynamic_job_slots is enabled, the daemon monitors memory usage of running jobs and calculates available slots based on free system memory. This helps prevent system overload by dynamically adjusting the number of concurrent jobs based on available resources.