百度MapReduce BMR

    Cluster(集群)

    新建cluster

    如下代码可以新建一个集群,集群包含1个master节点和2个core节点,且安装了Hive、Pig、HBase应用。请注意:参考下面样例代码时,需要修改log_uri参数指定BOS路径为您的账户可用的BOS路径。

    instance_groups = [
        BmrClient.instance_group(
            'Master',
            'bmr.g1.xlarge	',
            1,
            'ig-master'
        ),
        BmrClient.instance_group(
            'Core',
            'bmr.g1.xlarge	',
            2,
            'ig-core'
        )
    ]  # 构造请求体
    request_body = {
        'applications': [
            {
                "name": "hive",
                "version": "0.13.0"
            },
            {
                "name": "pig",
                "version": "0.11.0"
            },
            {
                "name": "hbase",
                "version": "0.98.0"
            }
        ],
        'auto_terminate': False,
        'log_uri': 'bos://tester01/sdk/',
        'name': 'sdk-cluster01',
        'service_ha_enabled': False,
        'safe_mode_enabled': False
    }
    
    try:
        response = bmr_client.create_cluster(
            'hadoop',
            '0.1.0',
            instance_groups,
            **request_body)
        cluster_id = response.cluster_id
    except BceHttpClientError as e:
        if isinstance(e.last_error, BceServerError):
            LOG.error('create_cluster failed. Response %s, code: %s, msg: %s'
                      % (e.last_error.status_code, e.last_error.code, e.last_error.message))
        else:
            LOG.error('create_cluster failed. Unknown exception: %s' % e)

    售卖的套餐类型请参考https://cloud.baidu.com/doc/BMR/s/6jwvxw85z

    列出全部cluster

    如下代码可以罗列出属于请求调用者的所有集群,用户可以通过配置查询参数max_keys来限制每次请求返回的集群数目:

    try:
        response = bmr_client.list_clusters(max_keys=5)
        LOG.debug('list total %s clusters' % len(response.clusters))
        # 输出各个集群的信息
        for cluster in response.clusters:
            LOG.debug('cluster: %s' % cluster)
    except BceHttpClientError as e:
        if isinstance(e.last_error, BceServerError):
            LOG.error('list_clusters failed. Response %s, code: %s, msg: %s'
                      % (e.last_error.status_code, e.last_error.code, e.last_error.message))
        else:
            LOG.error('list_clusters failed. Unknown exception: %s' % e)

    另外,也可以使用分页查询的方式来获取所有集群,通过配置max_keys来限制每页查询返回的集群数目:

    try:
        page = 1
        next_marker = None
        max_keys = 5
        is_truncated = True
        while is_truncated:
            response = bmr_client.list_clusters(marker=next_marker, max_keys=max_keys)
            LOG.debug('list %s clusters on Page %s' % (len(response.clusters), page))
            page += 1
            is_truncated = response.is_truncated
            next_marker = response.next_marker
    except BceHttpClientError as e:
        if isinstance(e.last_error, BceServerError):
            LOG.error('list_clusters failed. Response %s, code: %s, msg: %s'
                      % (e.last_error.status_code, e.last_error.code, e.last_error.message))
        else:
            LOG.error('list_clusters failed. Unknown exception: %s' % e)

    查询指定的cluster

    获取了集群ID后,可用如下代码查询指定集群的信息:

    try:
        response = bmr_client.get_cluster(cluster_id)
        # 输出集群的状态信息
        status = response.status.state
        LOG.debug('check cluster %s: status %s' % (cluster_id, status))
    except BceHttpClientError as e:
        if isinstance(e.last_error, BceServerError):
            LOG.error('get_cluster failed. Response %s, code: %s, msg: %s'
                      % (e.last_error.status_code, e.last_error.code, e.last_error.message))
        else:
            LOG.error('describe_cluster failed. Unknown exception: %s' % e)

    终止指定的cluster

    如下代码可以终止指定的集群:

    try:
        response = bmr_client.terminate_cluster(cluster_id)
        LOG.debug('terminate cluster %s: status %s' % (cluster_id, response.status))
    except BceHttpClientError as e:
        if isinstance(e.last_error, BceServerError):
            LOG.error('terminate_cluster failed. Response %s, code: %s, msg: %s'
                      % (e.last_error.status_code, e.last_error.code, e.last_error.message))
        else:
            LOG.error('terminate_cluster failed. Unknown exception: %s' % e)
    上一篇
    InstanceGroup(实例组)
    下一篇
    Step(作业)