百度MapReduce BMR

    作业操作接口

    添加作业

    接口描述

    向指定的集群批量添加自定义作业

    权限说明

    请求发起人需要具有合法的AccessKeyID和SecretAccessKey才能发起请求,请参考 鉴权认证

    注意事项

    如果请求中没有用户验证信息(即匿名访问),返回403 Forbidden,错误信息:AccessDenied

    请求结构

    POST /v{version}/cluster/{clusterId}/step?clientToken={clientToken} HTTP/1.1
    accept-encoding: gzip, deflate
    x-bce-date: {utc-date-string}
    host: bmr.bj.baidubce.com
    connection: keep-alive
    accept: */*
    content-type: application/json
    authorization: {bce-authorization-string}
    {
        "steps": [
            {
                "actionOnFailure": "Continue",
                "name": "my-mapreduce-streaming",
                "properties": {
                    "arguments": "-D mapreduce.job.reduces=20",
                    "input": "bos://mybucket/inputs/txt-1m",
                    "mapper": "cat",
                    "output": "bos://mybucket/outputs/streaming",
                    "reducer": "cat"
                },
                "type": "Streaming"
            },
            {
                "actionOnFailure": "Continue",
                "name": "my-java-program",
                "properties": {
                    "arguments": "bos://mybucket/inputs/java bos://mybucket/outputs/java",
                    "jar": "bos://mybucket/jars/my-java.jar",
                    "mainClass": "org.apache.hadoop.mapred.WordCount"
                },
                "type": "Java"
            }
        ]
    }

    请求头域

    除公共头域外,无其它特殊头域。

    请求参数

    参数名 参数类型 是否必须 参数位置 参数描述
    version String URL参数 API版本号,当前取值1
    clusterId String URL参数 待指定的集群ID,只有状态为RUNNING或WAITING的集群才能添加作业
    steps List\<StepConfig> RequestBody参数 需要添加的作业信息
    clientToken String Query参数 幂等性Token,是一个长度不超过64位的ASCII字符串

    响应头域

    除公共头域外,无其它特殊头域。

    响应参数

    参数名 参数类型 参数描述
    stepIds List\ 作业ID的集合,作业ID是系统生成的作业唯一标识

    请求示例

    POST /v1/cluster/0ce4f730-4af2-4f37-8fa2-b14f2f44e50e/step?clientToken=be31b98c-5e41-4838-9830-9be700de5a20 HTTP/1.1
    accept-encoding: gzip, deflate
    x-bce-date: 2015-03-24T13:02:00Z
    host: bmr.bj.baidubce.com
    connection: keep-alive
    accept: */*
    content-type: application/json
    authorization: bce-auth-v1/46bd9968a6194b4bbdf0341f2286ccce/2015-03-24T13:02:00Z/1800/host;x-bce-date/994014d96b0eb26578e039fa053a4f9003425da4bfedf33f4790882fb4c54903
    
    {
        "steps": [
            {
                "actionOnFailure": "Continue",
                "name": "my-mapreduce-streaming",
                "properties": {
                    "arguments": "-D mapreduce.job.reduces=20",
                    "input": "bos://mybucket/inputs/txt-1m",
                    "mapper": "cat",
                    "output": "bos://mybucket/outputs/streaming",
                    "reducer": "cat"
                },
                "type": "Streaming"
            },
            {
                "actionOnFailure": "Continue",
                "name": "my-java-program",
                "properties": {
                    "arguments": "bos://mybucket/inputs/java bos://mybucket/outputs/java",
                    "jar": "bos://mybucket/jars/my-java.jar",
                    "mainClass": "org.apache.hadoop.mapred.WordCount"
                },
                "type": "Java"
            }
        ]
    }

    响应示例

    HTTP/1.1 201 CREATED
    Transfer-Encoding: chunked
    x-bce-request-id: 73c4e74c-3101-4a00-bf44-fe246959c05e
    Cache-Control: no-cache
    Server: BWS
    Date: Tue, 24 Mar 2015 13:02:01 GMT
    Content-Type: application/json;charset=UTF-8
    
    {
        "stepIds": [
            "31fd0974-452b-491f-a3fc-c28b3b9d7046",
            "f99f669b-be45-48c9-8549-fd8dd473be0f"
        ]
    }

    查询作业列表

    接口描述

    获取指定集群的作业列表,支持分页查询

    权限说明

    请求发起人需要具有合法的AccessKeyID和SecretAccessKey才能发起请求,请参考 鉴权认证

    注意事项

    如果请求中没有用户验证信息(即匿名访问),返回403 Forbidden,错误信息:AccessDenied

    请求结构

    GET /v{version}/cluster/{clusterId}/step?marker={marker}&maxKeys={maxKeys} HTTP/1.1
    accept-encoding: gzip, deflate
    x-bce-date: {utc-date-string}
    host: bmr.bj.baidubce.com
    connection: keep-alive
    accept: */*
    content-type: application/json
    authorization: {bce-authorization-string}

    请求头域

    除公共头域外,无其它特殊头域。

    请求参数

    参数名 参数类型 是否必须 参数位置 参数描述
    version String URL参数 API版本号,当前取值1
    clusterId String URL参数 指定的集群ID
    marker String Query参数 批量获取列表的查询的起始位置,是一个由系统生成的字符串
    maxKeys Int Query参数 每页包含的最大数量,最大数量不能超过1000。大于1000的会被修正为1000。缺省值为1000

    响应头域

    除公共头域外,无其它特殊头域。

    响应参数

    参数名 参数类型 参数描述
    marker String 标记查询的起始位置
    isTruncated Boolean true表示后面还有数据,false表示已经是最后一页
    nextMarker String 获取下一页所需要传递的marker值。当isTruncated为false时,该域不出现
    steps List\<Step> 返回的作业列表

    请求示例

    GET /v1/cluster/0ce4f730-4af2-4f37-8fa2-b14f2f44e50e/step?marker=9e0c8cf1-690c-444a-8727-04241f3beaa5&maxKeys=30 HTTP/1.1
    accept-encoding: gzip, deflate
    x-bce-date: 2015-03-24T13:02:00Z
    host: bmr.bj.baidubce.com
    connection: keep-alive
    accept: */*
    content-type: application/json
    authorization: bce-auth-v1/46bd9968a6194b4bbdf0341f2286ccce/2015-03-24T13:02:00Z/1800/host;x-bce-date/994014d96b0eb26578e039fa053a4f9003425da4bfedf33f4790882fb4c54903

    响应示例

    HTTP/1.1 200 OK
    Transfer-Encoding: chunked
    x-bce-request-id: 73c4e74c-3101-4a00-bf44-fe246959c05e
    Cache-Control: no-cache
    Server: BWS
    Date: Tue, 24 Mar 2015 13:02:01 GMT
    Content-Type: application/json;charset=UTF-8
    
    {
        "isTruncated": false,
        "marker": "9e0c8cf1-690c-444a-8727-04241f3beaa5",
        "steps": [
            {
                "actionOnFailure": "Continue",
                "id": "6e0e826f-462c-4529-a5ef-b7d9d86e7741",
                "name": "my-java-program",
                "properties": {
                    "arguments": "bos://mybucket/inputs/java bos://mybucket/outputs/java",
                    "jar": "bos://mybucket/jars/my-java.jar",
                    "mainClass": "org.apache.hadoop.mapred.WordCount"
                },
                "status": {
                    "code": null,
                    "creationDateTime": "2014-08-06T11:00:30Z",
                    "endDateTime": null,
                    "message": null,
                    "startDateTime": "2014-08-06T11:02:55Z",
                    "state": "Running"
                },
                "type": "Java"
            },
            {
                "actionOnFailure": "Continue",
                "id": "f5540043-b1b1-4a52-95a7-6005cb0ebc64",
                "name": "my-mapreduce-streaming",
                "properties": {
                    "arguments": "-D mapreduce.job.reduces=20",
                    "input": "bos://mybucket/inputs/txt-1m",
                    "mapper": "cat",
                    "output": "bos://mybucket/outputs/streaming",
                    "reducer": "cat"
                },
                "status": {
                    "code": null,
                    "creationDateTime": "2014-08-06T11:00:05Z",
                    "endDateTime": "2014-08-06T11:02:37Z",
                    "message": null,
                    "startDateTime": "2014-08-06T11:00:33Z",
                    "state": "Completed"
                },
                "type": "Streaming"
            }
        ]
    }

    查询作业信息

    接口描述

    查询指定集群中的一个指定作业的信息

    权限说明

    请求发起人需要具有合法的AccessKeyID和SecretAccessKey才能发起请求,请参考 鉴权认证

    注意事项

    如果请求中没有用户验证信息(即匿名访问),返回403 Forbidden,错误信息:AccessDenied

    请求结构

    GET /v{version}/cluster/{clusterId}/step/{stepId} HTTP/1.1
    accept-encoding: gzip, deflate
    x-bce-date: {utc-date-string}
    host: bmr.bj.baidubce.com
    connection: keep-alive
    accept: */*
    content-type: application/json
    authorization: {bce-authorization-string}

    请求头域

    除公共头域外,无其它特殊头域。

    请求参数

    参数名 参数类型 是否必须 参数位置 参数描述
    version String URL参数 API版本号,当前取值1
    clusterId String URL参数 指定的集群ID
    stepId String URL参数 待查询的作业ID

    响应头域

    除公共头域外,无其它特殊头域。

    响应参数

    参数名 参数类型 参数描述
    actionOnFailure String 作业失败策略
    id String 作业ID,是一个定长字符串,只包含大小写字母、数字、连字号(-)和下划线(_)
    name String 作业名称
    properties Object 作业描述,具体内容由作业类型决定
    status StepStatus 作业状态信息
    type String 作业类型

    请求示例

    GET /v1/cluster/0ce4f730-4af2-4f37-8fa2-b14f2f44e50e/step/f5540043-b1b1-4a52-95a7-6005cb0ebc64 HTTP/1.1
    accept-encoding: gzip, deflate
    x-bce-date: 2015-03-24T13:02:00Z
    host: bmr.bj.baidubce.com
    connection: keep-alive
    accept: */*
    content-type: application/json
    authorization: bce-auth-v1/46bd9968a6194b4bbdf0341f2286ccce/2015-03-24T13:02:00Z/1800/host;x-bce-date/994014d96b0eb26578e039fa053a4f9003425da4bfedf33f4790882fb4c54903

    响应示例

    HTTP/1.1 200 OK
    Transfer-Encoding: chunked
    x-bce-request-id: 73c4e74c-3101-4a00-bf44-fe246959c05e
    Cache-Control: no-cache
    Server: BWS
    Date: Tue, 24 Mar 2015 13:02:01 GMT
    Content-Type: application/json;charset=UTF-8
    
    {
        "actionOnFailure": "Continue",
        "id": "f5540043-b1b1-4a52-95a7-6005cb0ebc64",
        "name": "my-mapreduce-streaming",
        "properties": {
            "arguments": "-D mapreduce.job.reduces=20",
            "input": "bos://mybucket/inputs/txt-1m",
            "mapper": "cat",
            "output": "bos://mybucket/outputs/streaming",
            "reducer": "cat"
        },
        "status": {
            "code": null,
            "creationDateTime": "2014-08-06T11:00:05Z",
            "endDateTime": "2014-08-06T11:02:37Z",
            "message": null,
            "startDateTime": "2014-08-06T11:00:33Z",
            "state": "Completed"
        },
        "type": "Streaming"
    }
    上一篇
    实例操作接口
    下一篇
    数据类型