Zookeeper? Curator ?

주키퍼를 이용해 master 를 점유하고, 작업을 분배하는걸 만들려고 했는데 

API 가 callback 구조로 만들어서 꽤 복잡하다.


그래서, 넷플릭스에서 만들고 아파치재단에 탑프로젝트로 올라온 curator 라는 프로젝트가 존재했다.

고수준의 주키퍼 API 랄까?


샘플코드와 오류!

* JmcSample.java
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.framework.recipes.leader.LeaderSelector;
import org.apache.curator.framework.recipes.leader.LeaderSelectorListener;
import org.apache.curator.framework.state.ConnectionState;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.apache.zookeeper.*;
import java.io.Closeable;
import java.io.IOException;

public class JmcSample {

    public static void main(String []argv) throws Exception {

        final String hostPorts = "11.11.11.11:2181/tost";
        String serverId = "===server0001===";
        CuratorFramework client;        
        client = CuratorFrameworkFactory.newClient(hostPorts, new ExponentialBackoffRetry(1000, 4));

        client.start();
        client.getZookeeperClient().blockUntilConnectedOrTimedOut();

        System.err.println("PERSISTENT > ");
        client.create().withMode(CreateMode.PERSISTENT).forPath("/"); // 즉 '/tost' 라는 폴더 생성
        client.close();

	}
}


* log

2017-08-17 12:11:32,209 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@749] - Received packet at server of unknown type 15

2017-08-17 12:11:32,209 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /10.10.10.10:58956 which had sessionid 0x15dbc19c3140034

2017-08-17 12:11:34,000 [myid:] - INFO  [SessionTracker:ZooKeeperServer@358] - Expiring session 0x15dbc19c3140032, timeout of 40000ms exceeded

2017-08-17 12:11:34,001 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@486] - Processed session termination for sessionid: 0x15dbc19c3140032

2017-08-17 12:11:48,000 [myid:] - INFO  [SessionTracker:ZooKeeperServer@358] - Expiring session 0x15dbc19c3140033, timeout of 40000ms exceeded

2017-08-17 12:11:48,001 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@486] - Processed session termination for sessionid: 0x15dbc19c3140033



해결법


이유는 단순했다. CDH 버전의 주키퍼와 버전을 맞추기위해 3.4.x 버전을 사용했는데, 버전충돌이 문제였다.

저렇게 하고 실행하니 성공한다. 허허허


* pom.xml

        <dependency>
            <groupId>org.apache.curator</groupId>
            <artifactId>curator-recipes</artifactId>
            <version>4.0.0</version>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.zookeeper</groupId>
                    <artifactId>zookeeper</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
            <version>3.4.10</version> 

        </dependency>


저작자표시 비영리 변경금지

실행했는데 아래와 같은 오류가 났다.


presto 의 프로세스가 떠있나 확인해 보니 데몬도 죽은상태.

이건 자바 버전이 낮았기 때문이었다. CDH 기본 내장된 자바버전은 1.7인데

presto 는 java 1.8 부터 지원을 한다.



* server.log

2017-06-08T13:27:28.145+0900    INFO    main    com.facebook.presto.server.PrestoServer ======== SERVER STARTED ========

2017-06-08T13:27:28.145+0900    ERROR   Announcer-0     io.airlift.discovery.client.Announcer   Cannot connect to discovery server for announce: Announcement failed with status code 404:

2017-06-08T13:27:28.145+0900    ERROR   Announcer-0     io.airlift.discovery.client.Announcer   Service announcement failed after 11.23ms. Next request will happen within 0.00s

2017-06-08T13:27:28.151+0900    ERROR   Announcer-1     io.airlift.discovery.client.Announcer   Service announcement failed after 3.12ms. Next request will happen within 1.00ms

2017-06-08T13:27:28.155+0900    ERROR   Announcer-2     io.airlift.discovery.client.Announcer   Service announcement failed after 2.32ms. Next request will happen within 2.00ms

2017-06-08T13:27:28.163+0900    ERROR   Announcer-3     io.airlift.discovery.client.Announcer   Service announcement failed after 3.28ms. Next request will happen within 4.00ms

2017-06-08T13:27:28.174+0900    ERROR   Announcer-1     io.airlift.discovery.client.Announcer   Service announcement failed after 2.89ms. Next request will happen within 8.00ms

2017-06-08T13:27:28.194+0900    ERROR   Announcer-2     io.airlift.discovery.client.Announcer   Service announcement failed after 3.62ms. Next request will happen within 16.00ms

2017-06-08T13:27:28.230+0900    ERROR   Announcer-3     io.airlift.discovery.client.Announcer   Service announcement failed after 3.57ms. Next request will happen within 32.00ms

...


jdk 1.8을 받아서 다시 실행하니 성공

저작자표시 비영리 변경금지

오라클이나 DBMS에서 자주 쓰던 형태로 쿼리를 hive에서 돌리면 이런 오류가 난다.


Unsupported SubQuery Expression 'keyword': Correlating expression cannot contain unqualified column references.



select

   *

from

  tb_sample

where

  keyword in (select keyword from tb_my_kwd)



그 이유는 메인테이블과 서브테이블의 keyword 명을 구분해줘야 하기 때문이다.

눈으로 보면 딱 알겠는데 hive 에서는 지정해줘야한다. ;;;;




select

   *

from

  tb_sample t

where

  t.keyword in (select keyword from tb_my_kwd)



저작자표시 비영리 변경금지

+ Recent posts

티스토리툴바