客户端加密实践
所有文档

          对象存储 BOS

          客户端加密实践

          背景概述

          客户端加密,是指由用户在本地自行完成文件的加密和解密过程,百度智能云对象存储不参与加密和解密过程,只负责文件的上传、存储、下载过程,明文密钥由用户自行保管在本地。客户端加密增强了文件安全性,即使文件意外泄露,别人也无法解密得到原始数据。但是由于用户需要自行负责保管密钥,如果明文密钥丢失,用户将无法获得原文件内容。本文档提供了一种客户端加密方案。

          加密原理

          1. 本地使用RSA算法生成非对称密钥(private_rsa_keypublic_rsa_key);
          2. 上传文件时,使用AES 256的CTR模式生成对称密钥(aes_key);
          3. 对于普通文件,使用对称密钥aes_key加密原始数据;使用公钥public_rsa_key加密aes_key生成encrypted_aes_key,作为object的meta,并上传文件到Bos。对于大文件,使用Bos的分块上传,每次取一个分块加密并上传,分块大小为16字节的整数倍,同样使用公钥public _rsa_key加密aes_key生成encrypted_aes_key,作为object的meta。
          4. 下载文件时,首先获取文件的meta信息,得到encrypted_aes_key,利用本地私钥private_rsa_key解密encrypted_aes_key得到aes_key。对于普通文件,下载加密后文件并使用aes_key解密得到原始数据。对于大文件,可以使用分块下载,使用Python SDK的函数get_object(bucket_name,object_key,[range_start,range_end])下载大文件的指定区间[range_start,range_end]中的字节,包含结尾位置range_end处的字节,其中0 ≤range_start≤range_end≤文件大小。

          注意:使用AES 256方式加密,AES算法加密时的块大小是128bits=16byte,因此在大文件分块上传时,每个分块大小应该是16字节的整数倍。

          加密原理图

          Python示例代码

          准备工作

          1.用户需要安装python SDk,请参考安装python SDK工具包

          2.执行下列命令,安装PyCrypto库。

          pip install pycrypto

          3.修改配置。修改示例代码中的HOST和AK、SK,作为本示例的配置项。

          示例代码如下

          #coding=utf-8
          
          '''
          The file aims to help client to encrypt data on python sdk with RSA algorithm  and symmetric encryption algorithm 
          '''
          
          import os
          import shutil
          import base64
          import random
          import string
          
          #引入配置文件和对象存储模块
          from baidubce.bce_client_configuration import BceClientConfiguration
          from baidubce.auth.bce_credentials import BceCredentials
          from baidubce import exception
          from baidubce.services import bos
          from baidubce.services.bos import canned_acl
          from baidubce.services.bos.bos_client import BosClient
          
          #引入Crypto加密模块
          from Crypto import Random
          from Crypto.Cipher  import AES
          from Crypto.Cipher  import PKCS1_OAEP
          from Crypto.PublicKey import RSA
          from Crypto.Util    import Counter
          
          #设置对称密钥长度为128bits
          _AES256_KEY_SIZE = 32
          #设置AES CTR模式的计数器Counter长度
          _COUNTER_BITS_LENGTH_AES_CTR = 8*16
          
          class CipherWithAES:
              # start为CTR计数器初始值
              def __init__(self, key= None, start= None):
                  if not key:
                      key = Random.new().read(_AES256_KEY_SIZE)
                  if not start:
                      start = random.randint(1,100)
                  self.key = key
                  self.start = start
                  #生成计数器
                  my_counter = Counter.new(_COUNTER_BITS_LENGTH_AES_CTR, initial_value=self.start)
                  #生成AES对象
                  self.cipher = AES.new(self.key, AES.MODE_CTR, counter = my_counter)
          
              #加密数据
              def encrypt(self, plaintext):
                  return self.cipher.encrypt(plaintext)
          
              #解密数据
              def decrypt(self, ciphertext):
                  return self.cipher.decrypt(ciphertext)
          
          class CipherWithRSA:
              # 输入参数为公钥文件名和私钥文件名
              def __init__(self, public_key_file_name = None, private_key_file_name = None):
                  self.public_key_file_name=public_key_file_name
                  self.private_key_file_name = private_key_file_name
                  if not self.public_key_file_name:
                      self.public_key_file_name = "rsa_public_key.pem"
                  if not self.private_key_file_name:
                      self.private_key_file_name = "rsa_private_key.pem"
                  #如果没有输入公钥和私钥文件,则本地产生RSA密钥对
                  if not (os.path.isfile(self.public_key_file_name) and os.path.isfile(self.private_key_file_name)):
                      self._generate_rsa_key()
                      return
                  #从文件读取公钥和私钥,并产生RSA对象
                  with open(self.public_key_file_name) as file_key:
                      public_key_obj_rsa = RSA.importKey(file_key.read())
                      self.encrypt_obj = PKCS1_OAEP.new(public_key_obj_rsa)
                  with open(self.private_key_file_name) as file_key:
                      private_key_obj_rsa = RSA.importKey(file_key.read())
                      self.decrypt_obj = PKCS1_OAEP.new(private_key_obj_rsa)
          
              #本地生成RSA密钥对
              def _generate_rsa_key(self):
                  private_key_obj_rsa = RSA.generate(2048)
                  public_key_obj_rsa = private_key_obj_rsa.publickey()
                  self.encrypt_obj = PKCS1_OAEP.new(public_key_obj_rsa)
                  self.decrypt_obj = PKCS1_OAEP.new(private_key_obj_rsa)
                  #将生成的密钥对存入本地
                  with open(self.public_key_file_name,"w") as file_export:
                      file_export.write(public_key_obj_rsa.exportKey())
                  with open(self.private_key_file_name,"w") as file_export:
                      file_export.write(private_key_obj_rsa.exportKey())
          
              #加密数据
              def encrypt(self,plaintext_key):
                  return self.encrypt_obj.encrypt(plaintext_key)
          
              #解密数据
              def decrypt(self,ciphertext_key):
                  return self.decrypt_obj.decrypt(ciphertext_key)
          #产生随机文件名
          def _random_string(length):
              return ''.join(random.choice(string.ascii_lowercase + string.digits) for _ in range(length))
          
          ################put super file#################
          #分块上传大文件到百度对象存储,"part_size"为分块大小,必须为16字节的整数倍
          def put_super_file(bos_client, bucket_name, super_object_key, super_file, part_size,cipher_rsa):
              """
              Put super file to baidu object storage by multipart uploading."part_size" must be multiple of 16 bytes
          
              :param bos_client:bos client
              :type bos_client:baidubce.services.bos.bos_client.BosClient
          
              :param bucket_name:None
              :type bucket_name:string
          
              :param super_object_key: destion object key of super file
              :type super_object_key: string
          
              :param super_file: super file name
              :type super_file:string
          
              :param part_size: size of part to upload once,"part_size" must be multiple of 16 bytes and more than 5MB
              :type: int
          
              :param cipher_rsa: encrypt symmetric key
              :type cipher_rsa: CipherWithRSA
          
              :return :**Http Response**
          
              """
              #1.initial
              try:
                  if not isinstance(bos_client,BosClient):
                      raise Exception("bos client is None!")
                  if not (bucket_name and super_object_key):
                      raise Exception("bucket or object is invalid!")
                  if not os.path.isfile(super_file):
                      raise Exception("source file is invalid!")
                  if not isinstance(cipher_rsa,CipherWithRSA):
                      raise Exception("cipher_rsa is invalid!")
              except Exception as e:
                  print e
                  exit()
              if not part_size:
                  part_size = 10*1024*1024
              #temp_file加密后的文件分块
              temp_file = _random_string(20)
              cipher_aes = CipherWithAES()
              #2.分块上传
              #分块上传分三步,第一步初始化,获取upload_id
              upload_id = bos_client.initiate_multipart_upload(
                  bucket_name = bucket_name,
                  key = super_object_key,
                  ).upload_id
              left_size = os.path.getsize(super_file)
              offset = 0
              part_number = 1
              part_list = []
              fs = open(super_file,"r")
              while left_size > 0:
                  if left_size < part_size:
                      part_size = left_size
                  # 读取分块并加密 
                  part_content = fs.read(part_size)
                  encrypted_part_content = cipher_aes.encrypt(part_content)
                  #分块加密后重新写入temp_file
                  with open(temp_file,"w") as ft:
                      ft.write(encrypted_part_content)
                  #分块上传第二步,上传分块
                  response = bos_client.upload_part_from_file(
                          bucket_name, super_object_key, upload_id, part_number, part_size, temp_file, offset)
                  left_size -= part_size
                  #保存part number和etag用于调用complete_multipart_upload()
                  part_list.append({
                      "partNumber": part_number,
                      "eTag": response.metadata.etag
                      })
                  part_number += 1
              os.remove(temp_file)
              fs.close()
              #使用公钥加密AES密钥,并作为object的meta
              user_metadata = {
                  "key": base64.b64encode(cipher_rsa.encrypt(str(cipher_aes.key))),
                  "start":base64.b64encode(cipher_rsa.encrypt(str(cipher_aes.start)))
              }
              #分块上传第三步,完成分块上传
              return bos_client.complete_multipart_upload(bucket_name, super_object_key, upload_id, part_list,user_metadata = user_metadata)
          
          ####################### Get super file#######################
          #获取大文件的分块,其中range_start, range_end分别为欲获取分块在文件中的起止位置,包含技术位置的字节
          def get_part_super_file(bos_client, bucket_name, super_object_key, range_start, range_end,cipher_rsa):
          
              """
              Get part of  super file from baidu object storage
          
              :param bos_client:bos client
              :type bos_client:baidubce.services.bos.bos_client.BosClient
          
              :param bucket_name:None
              :type bucket_name:string
          
              :param super_object_key: destion object key of super file
              :type super_object_key: string
          
              :param range_start: index of first bytes of part,minimum  value is 0
              :type super_file:int
          
              :param range_end: index of last bytes of part 
              :type: int
          
              :param cipher_rsa: dencrypt symmetric key
              :type cipher_rsa: CipherWithRSA
          
              :return :**Http Response**
          
              """
          
              try:
                  if not isinstance(bos_client,BosClient):
                      raise Exception("bos client is None!")
                  if not (bucket_name and super_object_key):
                      raise Exception("bucket or object is invalid!")
                  if not (range_start and range_end):
                      raise Exception("range is invalid!")
                  if not isinstance(cipher_rsa,CipherWithRSA):
                      raise Exception("cipher_rsa is invalid!")
              except Exception as e:
                  print e
                  exit()
              #1.对齐分块起始位置到16字节的整数倍
              left_offset = range_start%16
              right_offset = 15 -range_end%16
              test_range = [range_start-left_offset,range_end+right_offset]
              #2.获取object的meta
              response = bos_client.get_object_meta_data(bucket_name, super_object_key)
              #使用本地公钥来解密AES密钥密文,得到AES密钥明文
              download_aes_key = base64.b64decode(getattr(response.metadata,"bce_meta_key"))
              download_aes_start = base64.b64decode(str(getattr(response.metadata,"bce_meta_start")))
              aes_key = cipher_rsa.decrypt(download_aes_key)
              aes_start = cipher_rsa.decrypt(download_aes_start)
              #根据欲获取分块的起始位置调整计数器初始值
              offset_start = int(aes_start)+range_start/16
              cipher_aes = CipherWithAES(aes_key,int(offset_start))
              #3. 下载分块密文数据并使用AES密钥解密
              response = bos_client.get_object(bucket_name, super_object_key,test_range)
              download_content = response.data.read()
              plaintext_content = cipher_aes.decrypt(download_content)
              #截取用户指定区段的明文数据并返回
              return plaintext_content[left_offset:range_end-range_start+left_offset+1]
          
          #############  put common file ####################
          #加密上传普通文件
          def put_common_file(bos_client, bucket_name, object_key, file_name, cipher_rsa):
              """
              Put file to baidu object storage
          
              :param bos_client:bos client
              :type bos_client:baidubce.services.bos.bos_client.BosClient
          
              :param bucket_name:None
              :type bucket_name:string
          
              :param object_key: destion object key of file
              :type object_key: string
          
              :param file_name: source file name
              :type file_name:string
          
              :param cipher_rsa: encrypt symmetric key
              :type cipher_rsa: CipherWithRSA
          
              :return :**Http Response**
          
              """
          
              try:
                  if not isinstance(bos_client,BosClient):
                      raise Exception("bos client is None!")
                  if not (bucket_name and object_key):
                      raise Exception("bucket or object is invalid!")
                  if not os.path.isfile(file_name):
                      raise Exception("file name  is invalid!")
                  if not isinstance(cipher_rsa,CipherWithRSA):
                      raise Exception("cipher_rsa is invalid!")
              except Exception as e:
                  print e
                  exit()
              temp_file = _random_string(20)
              #读取欲上传文件数据
              content=""
              with open(file_name,"r") as fp:
                  content = fp.read()
              cipher_aes = CipherWithAES()
              #加密数据并写到temp_file
              encrypt_content = cipher_aes.encrypt(content)
              with  open(temp_file,"w") as ft:
                  ft.write(encrypt_content)
              cipher_rsa = CipherWithRSA()
              #加密AES密钥作为Object的meta
              user_metadata = {
                      "key": base64.b64encode(cipher_rsa.encrypt(str(cipher_aes.key))),
                      "start":base64.b64encode(cipher_rsa.encrypt(str(cipher_aes.start)))
                      }
              #上传加密后文件
              response = bos_client.put_object_from_file(bucket = bucket_name,
                      key = object_key,
                      file_name = temp_file,
                      user_metadata = user_metadata)
              os.remove(temp_file)
              return response
          
          ################get common file#####################
          #下载加密的普通文件
          def get_common_file(bos_client, bucket_name, object_key, des_file, cipher_rsa):
          
              """
              Put file to baidu object storage
          
              :param bos_client:bos client
              :type bos_client:baidubce.services.bos.bos_client.BosClient
          
              :param bucket_name:None
              :type bucket_name:string
          
              :param object_key: destion object key of file
              :type object_key: string
          
              :param des_file: destination file name
              :type des_file:string
          
              :param cipher_rsa: dencrypt symmetric key
              :type cipher_rsa: CipherWithRSA
          
              :return :**Http Response**
          
              """
              try:
                  if not isinstance(bos_client,BosClient):
                      raise Exception("bos client is None!")
                  if not (bucket_name and object_key):
                      raise Exception("bucket or object is invalid!")
                  if not des_file:
                      raise Exception("destination file is invalid!")
                  if not isinstance(cipher_rsa,CipherWithRSA):
                      raise Exception("cipher_rsa is invalid!")
              except Exception as e:
                  print e
                  exit()
              #下载获取meta
              response = bos_client.get_object_meta_data(bucket_name, object_key)
              download_aes_key = base64.b64decode(getattr(response.metadata,"bce_meta_key"))
              download_aes_start = base64.b64decode(str(getattr(response.metadata,"bce_meta_start")))
              #下载加密数据到本地
              download_content = bos_client.get_object_as_string(bucket_name, object_key)
              #解密得到AES密钥明文
              aes_key = cipher_rsa.decrypt(download_aes_key)
              aes_start = cipher_rsa.decrypt(download_aes_start)
          
              cipher_aes = CipherWithAES(aes_key,int(aes_start))
              plaintext_content = cipher_aes.decrypt(download_content)
              with open(des_file,"w") as fd:
                  fd.write(plaintext_content)
          
          if __name__ == "__main__":
          	#以北京地区对象存储为例,替换AK、SK、bucket_name为用户的数据
              HOST = 'bj.bcebos.com'
              AK = 'Your_Access_Key'
              SK = 'Your_Secret_Access_Key'
          	bucket_name = "Your-Bucket-Name"
              super_file= "super_file"
              super_object_key = "my-super-object"
              #获取bos client
              config = BceClientConfiguration(credentials=BceCredentials(AK, SK), endpoint=HOST)
              bos_client = BosClient(config)
          
              #1.1上传大文件
              #设置分块大小
              part_size = 10*1024*1024
              cipher_rsa = CipherWithRSA()
              #分块上传大文件
              put_super_file(bos_client, bucket_name, super_object_key, super_file, part_size,cipher_rsa)
              #1.2 获取大文件分块
              #设置欲获取的分块起始位置,建议设置成16字节的整数倍
              range_start = 16*10
              range_end = 16*11-1
              #获取明文数据块
              result = get_part_super_file(bos_client, bucket_name, super_object_key, range_start, range_end,cipher_rsa)
              print "#"*20
              print result
              print "#"*20
              print "length:",len(result)
          
              #2.1上传普通文件
              object_key = "myobject"
              source_file = "myobject.txt"
              put_common_file(bos_client,bucket_name,object_key,source_file,cipher_rsa)
              #2.2 下载普通文件
              des_file = "des_myobject.txt"
              get_common_file(bos_client,bucket_name,object_key,des_file,cipher_rsa)
          上一篇
          如何解决浏览器跨域CORS问题
          下一篇
          通过安卓SDK使用HTTPDNS服务