Spring Quartz Clustering

,

이슈

운영중인 서버에서 스케줄링 작업을 하는게 있었는데 실행이 3번 중복 실행되는 문제가 발생하였다.
문자를 발송하는 작업이였는데 3번 발송됨

분석

실행환경은 Spring(4.0.0) + Quartz(1.8.5) 를 사용하고 있었다.
기존에 스프링 quartz bean 설정은 다음과 같았다.
MethodInvokingJobDetailFactoryBean을 사용하여 스프링 빈의 메소드를 바로 실행시킨다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
<bean id="smsSendUserCollectRequest" class="org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean">
    <property name="targetObject" ref="smsSendUserCollectScheduling" />
    <property name="targetMethod" value="smsSendUserCollectCollectMethod" />
    <property name="concurrent" value="false" />
</bean>
<bean id="smsSendUserCollectRequestTrigger" class="org.springframework.scheduling.quartz.CronTriggerFactoryBean">
    <property name="jobDetail" ref="smsSendUserCollectRequest" />
    <property name="cronExpression" value="0 0 5 * * ?" />
</bean>
<bean id="smsSendUserCollectCollectRequestScheduler"
    class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
    <property name="triggers">
        <list>
            <ref bean="smsSendUserCollectRequestTrigger" />				
        </list>
    </property>
</bean>

cron trigger 표현식을 보면 “0 0 5 * * ?” 5시에 한번만 실행되는걸 볼 수 있다.
스케줄러 자체 설정엔 문제가 없었지만 중복실행되는 이유는
WAS가 이중화되어 있었기 때문이였다. JEUS WAS를 사용하는데 3개의 container로 나뉘어져있었고 3개의 container에서 모두 실행되고 있었던 것이다.

quart-clustering

다행히 quartz lib에서 clustering을 지원하고 있어서 생각보다 어렵지 않게 문제를 해결할 수 있었다.
클러스터링을 적용하려면 스프링쿼츠의 SchedulerFactoryBean 설정을 변경해주어야한다.

설정 변경

  1. JobDetailBean로 변경

기존 설정에서는 org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean을 사용하여 스프링 빈의 메소드를 바로 실행했는데
이방식은 클러스터링이되지 않기 때문에 org.springframework.scheduling.quartz.JobDetailBean 방식으로 변경해야했다.
JobDetailBean의 JobClass는 QuartzJobBean 클래스를 상속하여 executeInternal 메소드를 오버라이드 해야한다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
@Service("smsSendUserCollectRequest")
public class smsSendUserCollectScheduling extends QuartzJobBean {
	@Autowired
	private SmsService smsService;
	
	@Override
	protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
		try {					
			doTask();
		} catch (Exception e) {
			e.printStackTrace();
			throw new JobExecutionException(e);
		}
	}

    public void doTask() {
        //do something
    }
}

  1. 스프링 빈 설정 변경
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<bean id="smsSendUserCollectRequest" class="org.springframework.scheduling.quartz.JobDetailBean">
	<property name="jobClass" value="io.timpac.smsSendUserCollectScheduling" />
</bean>
<bean id="smsSendUserCollectRequestTrigger" class="org.springframework.scheduling.quartz.CronTriggerFactoryBean">
	<property name="jobDetail" ref="smsSendUserCollectRequest" />
	<property name="cronExpression" value="0 0 5 * * ?" />
</bean>
<bean id="ClusterScheduler" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
	<property name="triggers">
		<list>
			<ref bean="smsSendUserCollectRequestTrigger" />		
		</list>
	</property>
	<property name="dataSource" ref="dataSource"/>
	<property name="configLocation" value="classpath:quartz.properties" />
  	</bean>

  1. quartz.properties 속성 설정
1
2
3
4
5
6
7
8
9
10
11
12
13
org.quartz.jobStore.useProperties=false
org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.dataSource=dataSource
org.quartz.jobStore.tablePrefix=QRTZ_
org.quartz.jobStore.isClustered=true
org.quartz.jobStore.clusterCheckinInterval=20000
org.quartz.jobStore.misfireThreshold=60000
org.quartz.scheduler.instanceId=AUTO
org.quartz.scheduler.instanceName=MyClusteredScheduler
org.quartz.threadPool.class=org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount=4
org.quartz.threadPool.threadPriority=5

각 속성의 설정정보는 아래링크 참조
http://www.quartz-scheduler.org/documentation/quartz-1.8.6/configuration/ConfigJDBCJobStoreClustering


  1. database table 생성
    클러스터링 동기화를 위한 데이터베이스 테이블을 생성해준다.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
CREATE TABLE qrtz_job_details
  (
    JOB_NAME  VARCHAR2(200) NOT NULL,
    JOB_GROUP VARCHAR2(200) NOT NULL,
    DESCRIPTION VARCHAR2(250) NULL,
    JOB_CLASS_NAME   VARCHAR2(250) NOT NULL, 
    IS_DURABLE VARCHAR2(1) NOT NULL,
    IS_VOLATILE VARCHAR2(1) NOT NULL,
    IS_STATEFUL VARCHAR2(1) NOT NULL,
    REQUESTS_RECOVERY VARCHAR2(1) NOT NULL,
    JOB_DATA BLOB NULL,
    PRIMARY KEY (JOB_NAME,JOB_GROUP)
);
CREATE TABLE qrtz_job_listeners
  (
    JOB_NAME  VARCHAR2(200) NOT NULL, 
    JOB_GROUP VARCHAR2(200) NOT NULL,
    JOB_LISTENER VARCHAR2(200) NOT NULL,
    PRIMARY KEY (JOB_NAME,JOB_GROUP,JOB_LISTENER),
    FOREIGN KEY (JOB_NAME,JOB_GROUP) 
	REFERENCES QRTZ_JOB_DETAILS(JOB_NAME,JOB_GROUP)
);
CREATE TABLE qrtz_triggers
  (
    TRIGGER_NAME VARCHAR2(200) NOT NULL,
    TRIGGER_GROUP VARCHAR2(200) NOT NULL,
    JOB_NAME  VARCHAR2(200) NOT NULL, 
    JOB_GROUP VARCHAR2(200) NOT NULL,
    IS_VOLATILE VARCHAR2(1) NOT NULL,
    DESCRIPTION VARCHAR2(250) NULL,
    NEXT_FIRE_TIME NUMBER(13) NULL,
    PREV_FIRE_TIME NUMBER(13) NULL,
    PRIORITY NUMBER(13) NULL,
    TRIGGER_STATE VARCHAR2(16) NOT NULL,
    TRIGGER_TYPE VARCHAR2(8) NOT NULL,
    START_TIME NUMBER(13) NOT NULL,
    END_TIME NUMBER(13) NULL,
    CALENDAR_NAME VARCHAR2(200) NULL,
    MISFIRE_INSTR NUMBER(2) NULL,
    JOB_DATA BLOB NULL,
    PRIMARY KEY (TRIGGER_NAME,TRIGGER_GROUP),
    FOREIGN KEY (JOB_NAME,JOB_GROUP) 
	REFERENCES QRTZ_JOB_DETAILS(JOB_NAME,JOB_GROUP) 
);
CREATE TABLE qrtz_simple_triggers
  (
    TRIGGER_NAME VARCHAR2(200) NOT NULL,
    TRIGGER_GROUP VARCHAR2(200) NOT NULL,
    REPEAT_COUNT NUMBER(7) NOT NULL,
    REPEAT_INTERVAL NUMBER(12) NOT NULL,
    TIMES_TRIGGERED NUMBER(10) NOT NULL,
    PRIMARY KEY (TRIGGER_NAME,TRIGGER_GROUP),
    FOREIGN KEY (TRIGGER_NAME,TRIGGER_GROUP) 
	REFERENCES QRTZ_TRIGGERS(TRIGGER_NAME,TRIGGER_GROUP)
);
CREATE TABLE qrtz_cron_triggers
  (
    TRIGGER_NAME VARCHAR2(200) NOT NULL,
    TRIGGER_GROUP VARCHAR2(200) NOT NULL,
    CRON_EXPRESSION VARCHAR2(120) NOT NULL,
    TIME_ZONE_ID VARCHAR2(80),
    PRIMARY KEY (TRIGGER_NAME,TRIGGER_GROUP),
    FOREIGN KEY (TRIGGER_NAME,TRIGGER_GROUP) 
	REFERENCES QRTZ_TRIGGERS(TRIGGER_NAME,TRIGGER_GROUP)
);
CREATE TABLE qrtz_blob_triggers
  (
    TRIGGER_NAME VARCHAR2(200) NOT NULL,
    TRIGGER_GROUP VARCHAR2(200) NOT NULL,
    BLOB_DATA BLOB NULL,
    PRIMARY KEY (TRIGGER_NAME,TRIGGER_GROUP),
    FOREIGN KEY (TRIGGER_NAME,TRIGGER_GROUP) 
        REFERENCES QRTZ_TRIGGERS(TRIGGER_NAME,TRIGGER_GROUP)
);
CREATE TABLE qrtz_trigger_listeners
  (
    TRIGGER_NAME  VARCHAR2(200) NOT NULL, 
    TRIGGER_GROUP VARCHAR2(200) NOT NULL,
    TRIGGER_LISTENER VARCHAR2(200) NOT NULL,
    PRIMARY KEY (TRIGGER_NAME,TRIGGER_GROUP,TRIGGER_LISTENER),
    FOREIGN KEY (TRIGGER_NAME,TRIGGER_GROUP) 
	REFERENCES QRTZ_TRIGGERS(TRIGGER_NAME,TRIGGER_GROUP)
);
CREATE TABLE qrtz_calendars
  (
    CALENDAR_NAME  VARCHAR2(200) NOT NULL, 
    CALENDAR BLOB NOT NULL,
    PRIMARY KEY (CALENDAR_NAME)
);
CREATE TABLE qrtz_paused_trigger_grps
  (
    TRIGGER_GROUP  VARCHAR2(200) NOT NULL, 
    PRIMARY KEY (TRIGGER_GROUP)
);
CREATE TABLE qrtz_fired_triggers 
  (
    ENTRY_ID VARCHAR2(95) NOT NULL,
    TRIGGER_NAME VARCHAR2(200) NOT NULL,
    TRIGGER_GROUP VARCHAR2(200) NOT NULL,
    IS_VOLATILE VARCHAR2(1) NOT NULL,
    INSTANCE_NAME VARCHAR2(200) NOT NULL,
    FIRED_TIME NUMBER(13) NOT NULL,
    PRIORITY NUMBER(13) NOT NULL,
    STATE VARCHAR2(16) NOT NULL,
    JOB_NAME VARCHAR2(200) NULL,
    JOB_GROUP VARCHAR2(200) NULL,
    IS_STATEFUL VARCHAR2(1) NULL,
    REQUESTS_RECOVERY VARCHAR2(1) NULL,
    PRIMARY KEY (ENTRY_ID)
);
CREATE TABLE qrtz_scheduler_state 
  (
    INSTANCE_NAME VARCHAR2(200) NOT NULL,
    LAST_CHECKIN_TIME NUMBER(13) NOT NULL,
    CHECKIN_INTERVAL NUMBER(13) NOT NULL,
    PRIMARY KEY (INSTANCE_NAME)
);
CREATE TABLE qrtz_locks
  (
    LOCK_NAME  VARCHAR2(40) NOT NULL, 
    PRIMARY KEY (LOCK_NAME)
);
INSERT INTO qrtz_locks values('TRIGGER_ACCESS');
INSERT INTO qrtz_locks values('JOB_ACCESS');
INSERT INTO qrtz_locks values('CALENDAR_ACCESS');
INSERT INTO qrtz_locks values('STATE_ACCESS');
INSERT INTO qrtz_locks values('MISFIRE_ACCESS');
create index idx_qrtz_j_req_recovery on qrtz_job_details(REQUESTS_RECOVERY);
create index idx_qrtz_t_next_fire_time on qrtz_triggers(NEXT_FIRE_TIME);
create index idx_qrtz_t_state on qrtz_triggers(TRIGGER_STATE);
create index idx_qrtz_t_nft_st on qrtz_triggers(NEXT_FIRE_TIME,TRIGGER_STATE);
create index idx_qrtz_t_volatile on qrtz_triggers(IS_VOLATILE);
create index idx_qrtz_ft_trig_name on qrtz_fired_triggers(TRIGGER_NAME);
create index idx_qrtz_ft_trig_group on qrtz_fired_triggers(TRIGGER_GROUP);
create index idx_qrtz_ft_trig_nm_gp on qrtz_fired_triggers(TRIGGER_NAME,TRIGGER_GROUP);
create index idx_qrtz_ft_trig_volatile on qrtz_fired_triggers(IS_VOLATILE);
create index idx_qrtz_ft_trig_inst_name on qrtz_fired_triggers(INSTANCE_NAME);
create index idx_qrtz_ft_job_name on qrtz_fired_triggers(JOB_NAME);
create index idx_qrtz_ft_job_group on qrtz_fired_triggers(JOB_GROUP);
create index idx_qrtz_ft_job_stateful on qrtz_fired_triggers(IS_STATEFUL);
create index idx_qrtz_ft_job_req_recovery on qrtz_fired_triggers(REQUESTS_RECOVERY);

결과

서버에 적용하고 모니터링 해본 결과 하나의 인스턴스만 실행되는 것을 확인할 수 있었다.
분산서버 환경이나 이중화된 WAS환경에서 Quartz를 적용하려면 꼭 클러스터링을 해 주어야 중복실행을 막을 수 있다.

주의 사항

  • 설정변경시 qrtz_* 테이블 삭제
    quartz trigger나 job 설정을 변경할 경우 생성했던 qrtz_* 테이블들의 정보도 지워준 후에 서버를 재시작하는게 깔끔하다.
    테이블에 trigger나 job 정보들이 남아있어서 중복실행되거나 에러를 발생시키는 경우가 있었다.
1
2
3
4
5
6
7
8
9
10
11
delete from qrtz_job_listeners;
delete from qrtz_trigger_listeners;
delete from qrtz_fired_triggers;
delete from qrtz_simple_triggers;
delete from qrtz_cron_triggers;
delete from qrtz_blob_triggers;
delete from qrtz_triggers;
delete from qrtz_job_details;
delete from qrtz_calendars;
delete from qrtz_paused_trigger_grps;
delete from qrtz_scheduler_state;
  • Quartz JobClass에서 Spring bean injection
    실행시킬 Job 클래스에서 @Autowired 등으로 스프링 빈을 주입받아 사용하고 싶을 때가 있다.
1
2
3
4
5
6
7
8
9
@Service("smsSendUserCollectRequest")
public class smsSendUserCollectScheduling extends QuartzJobBean {
	@Autowired
	private SmsService smsService;

    protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
        smsService.send();  // NullPointerException 발생!
    }
}

위와 같이 사용하니까 NullPointerException이 발생했다.
아마도 쿼츠에서는 스프링 빈을 가져오지 않고 프록시 클래스를 사용해 메소드를 호출하기 때문에 의존성 주입이 안된 채로 실행되는게 아닐까 생각한다.
나는 메소드안에서 직접 의존성을 주입해주는 방식으로 해결했다.
쿼츠 잡클래스에서 빈을 주입받으려면 SchedulerFactoryBean에 applicationContextSchedulerContextKey 설정을 해야한다.

1
2
3
4
5
<bean id="ClusterScheduler" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
    ...
    <property name="applicationContextSchedulerContextKey" value="applicationContext"/> 
    ...
</bean>
1
2
3
4
5
6
7
8
9
10
11
12
@Service("smsSendUserCollectRequest")
public class smsSendUserCollectScheduling extends QuartzJobBean {
	@Autowired
	private SmsService smsService;

    protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
        ApplicationContext applicationContext = (ApplicationContext) context.getScheduler().getContext().get("applicationContext");
		this.smsService = applicationContext.getBean(SmsService.class);

        smsService.send(); 
    }
}

참조

http://www.quartz-scheduler.org/documentation/quartz-1.8.6/configuration/ConfigJDBCJobStoreClustering
https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#io.quartz
https://junhyunny.github.io/spring-mvc/quartz-clustering-in-spring-mvc/