稀疏鏡像在OpenHarmony上的應(yīng)用

開源項目 OpenHarmony是每個人的 OpenHarmony

陳甲印

鴻湖萬聯(lián)高級技術(shù)專家

鴻湖萬聯(lián)產(chǎn)品推薦官

一、稀疏鏡像升級背景

常用系統(tǒng)鏡像格式為原始鏡像，即RAW格式。鏡像體積比較大，在燒錄固件或者升級固件時比較耗時，而且在移動設(shè)備升級過程時比較耗費流量。為此，將原始鏡像用稀疏描述，可以大大地縮減鏡像體積，省時省流量。

二、稀疏鏡像原理

1、稀疏鏡像的概念 原始鏡像：即raw image，完整的ext4分區(qū)鏡像，包含很多全零的無效填充區(qū) 稀疏鏡像：即sparse image，將raw ext4進(jìn)行稀疏描述，因此尺寸比較小，制作目錄有多少文件就計算多少，沒有全零填充2、稀疏鏡像格式 ? 稀疏鏡像數(shù)據(jù)格式：首先是sparse_header占用28byte，然后是12byte的chunk_header，同樣這chunk_header的類型決定了后面跟著的數(shù)據(jù)，如果讀到數(shù)據(jù)是0xCAC1意味著后面是本身的raw_data，如果是0xCAC3，則后面num為0，接著再0xCAC2意味著后面填充4byte的內(nèi)容。 ?

三、實現(xiàn)稀疏鏡像升級方案

版本基線：

OpenAtom OpenHarmony（以下簡稱“OpenHarmony”） 3.1 Release

代碼路徑：

https://gitee.com/openharmony/docs/blob/master/zh-cn/release-notes/OpenHarmony-v3.1-release.md

1、稀疏鏡像燒錄（1）生成稀疏格式鏡像有2種方法可以生成稀疏鏡像：1）修改文件build/ohos_var.gni中，sparse_image=true ?2）編譯命令增加--sparse-image字段，如./build.sh --product-name=xxx --sparse-image（2）增加稀疏格式轉(zhuǎn)換工具在目錄build/ohos/images/mkimage中增加文件img2simg，該工具用于編譯完成后將raw鏡像轉(zhuǎn)換為sparse格式，并設(shè)置權(quán)限為777。（3）編譯后的鏡像對比 ?編譯出的鏡像格式為sparse格式，鏡像大小相比raw格式明顯變小。（4）燒錄稀疏鏡像燒錄稀疏鏡像方法和燒錄原始鏡像方法一致。稀疏鏡像本身是不能直接掛載的，在燒錄過程中通過uboot將稀疏格式鏡像還原為原始鏡像，然后寫到磁盤中，系統(tǒng)啟動后可掛載對應(yīng)的鏡像。 2、稀疏鏡像升級升級包采用稀疏鏡像制作。（1）修改升級包制作工具官方升級包工具不支持生成稀疏鏡像的升級包，修改升級包工具，生成稀疏格式的升級包。.aseupdatepackaging_toolsimage_class.py ?按照上圖所示注釋代碼（2）生成稀疏鏡像升級包和全量鏡像升級包制作方法一致。

參考：

https://gitee.com/openharmony/docs/blob/master/zh-cn/device-dev/subsystems/subsys-ota-guide.md#%E6%A0%87%E5%87%86%E7%B3%BB%E7%BB%9F%E5%8D%87%E7%BA%A7%E5%8C%85%E5%88%B6%E4%BD%9C

（3）適配updater組件中稀疏鏡像功能●增加寫稀疏鏡像分支.aseupdateupdaterservicesapplypatchdata_writer.cpp寫數(shù)據(jù)函數(shù)CreateDataWriter增加寫稀疏鏡像分支

case WRITE_SPARSE:
{
    std::unique_ptr writer(std::make_unique(partitionName));
    return std::move(writer);
}

●增加稀疏鏡像類聲明.aseupdateupdaterservicesapplypatch aw_writer.h增加稀疏鏡像類聲明及相關(guān)變量定義

typedefstructsparse_header{
 uint32_tmagic;  /* 0xed26ff3a */
 uint16_tmajor_version;/* (0x1) - reject images with higher major versions */
 uint16_tminor_version;/* (0x0) - allow images with higer minor versions */
 uint16_tfile_hdr_sz; /* 28 bytes for first revision of the file format */
 uint16_tchunk_hdr_sz; /* 12 bytes for first revision of the file format */
 uint32_tblk_sz;  /* block size in bytes, must be a multiple of 4 (4096) */
 uint32_ttotal_blks;/* total blocks in the non-sparse output image */
 uint32_ttotal_chunks; /* total chunks in the sparse input image */
 uint32_timage_checksum;/* CRC32 checksum of the original data, counting "don't care" */
        /* as 0. Standard 802.3 polynomial, use a Public Domain */
        /* table implementation */
} sparse_header_t;

#defineSPARSE_HEADER_MAGIC 0xed26ff3a

#defineCHUNK_TYPE_RAW   0xCAC1
#defineCHUNK_TYPE_FILL   0xCAC2
#defineCHUNK_TYPE_DONT_CARE  0xCAC3
#defineCHUNK_TYPE_CRC32  0xCAC4

typedefstructchunk_header{
 uint16_tchunk_type;/* 0xCAC1 -> raw; 0xCAC2 -> fill; 0xCAC3 -> don't care */
 uint16_treserved1;
 uint32_tchunk_sz; /* in blocks in output image */
 uint32_ttotal_sz; /* in bytes of chunk input file including chunk header and data */
} chunk_header_t;

classSparseWriter: publicDataWriter{
public:
  virtualboolWrite(constuint8_t*addr, size_tlen, WriteModemode, conststd::string&partitionName);

  explicitSparseWriter(conststd::stringpartitionName) : offset_(0), fd_(-1), partitionName_(partitionName) {}

  virtual~SparseWriter()
  {
    offset_= 0;
    if(fd_> 0) {
      fsync(fd_);
      close(fd_);
    }
    fd_= -1;
  }
private:
  intWriteInternal(intfd, constuint8_t*data, size_tlen, conststd::string&partitionName);

  SparseWriter(constSparseWriter&) = delete;

  constSparseWriter&operator=(constSparseWriter&) = delete;
  off64_toffset_;
  intfd_;
  std::string partitionName_;
};

●增加稀疏鏡像類實現(xiàn).aseupdateupdaterservicesapplypatch aw_writer.cpp增加稀疏鏡像類實現(xiàn)及相關(guān)變量定義，原有代碼不變

bool SparseWriter::Write(const uint8_t *addr, size_t len, WriteMode mode, const std::string &partitionName)
{
  if (addr == nullptr) {
    LOG(ERROR) << "SparseWriter: invalid address.";
    return false;
  }
  if (len == 0) {
    LOG(INFO) << "SparseWriter: write length is 0, skip.";
    return false;
  }
  if (fd_ < 0) {
    fd_ = OpenPartition(partitionName_);
    if (fd_ < 0) {
      return false;
    }
  }

  UPDATER_CHECK_ONLY_RETURN(WriteInternal(fd_, addr, len, partitionName_) >= 0, return false);
  return true;
}


int SparseWriter::WriteInternal(int fd, const uint8_t *data, size_t len, const std::string &partitionName)
{
  uint32_t written = 0;
  sparse_header_t *sparse_header;
  chunk_header_t *chunk_header;
  unsigned int chunk;
  void *membuf = NULL;
  uint32_t *fill_buf = NULL;
  uint32_t fill_val;
  uint32_t bytes_written = 0;
  uint32_t total_bytes = 0;
  uint32_t blk = 0;
  uint32_t chunk_data_sz = 0;
  uint32_t blkcnt = 0;
  uint32_t blks = 0;
  uint32_t total_blocks = 0;
  uint32_t addr_offset = 0;
  uint32_t fill_buf_num_blks = 0;


  uint32_t block_size = 4096;
  uint32_t block_count = 524288;
  uint32_t i;
  uint32_t j;
  int ret = lseek64(fd, offset_, SEEK_SET);
  UPDATER_FILE_CHECK(ret != -1, "RawWriter: failed to seek file to " << offset_, return -1);
  fill_buf_num_blks = CONFIG_FASTBOOT_FLASH_FILLBUF_SIZE / block_size;
  LOG(INFO) << "WriteInternal offset_ " << offset_;
  /* Read and skip over sparse image header */
  sparse_header = (sparse_header_t *)data;
  data += sparse_header->file_hdr_sz;
  if (sparse_header->file_hdr_sz > sizeof(sparse_header_t)) {
    /*
    * Skip the remaining bytes in a header that is longer than
    * we expected.
    */
    data += (sparse_header->file_hdr_sz - sizeof(sparse_header_t));
  }
  LOG(INFO) << "=== Sparse Image Header ===";
  LOG(INFO) << "magic: " ?<< sparse_header->magic;
  LOG(INFO) << "major_version: " << sparse_header->major_version;
  LOG(INFO) << "minor_version: " << sparse_header->minor_version;
  LOG(INFO) << "file_hdr_sz: " << sparse_header->file_hdr_sz;
  LOG(INFO) << "chunk_hdr_sz: " << sparse_header->chunk_hdr_sz;
  LOG(INFO) << "blk_sz: " << sparse_header->blk_sz;
  LOG(INFO) << "total_blks: " << sparse_header->total_blks;
  LOG(INFO) << "total_chunks: " << sparse_header->total_chunks;


  LOG(INFO) << "Flashing Sparse Image";
  blk = 0;
  for (chunk = 0; chunk < sparse_header->total_chunks; chunk++) {
    /* Read and skip over chunk header */
    chunk_header = (chunk_header_t *)data;
    data += sizeof(chunk_header_t);
    if (chunk_header->chunk_type != CHUNK_TYPE_RAW) 
    {
      LOG(INFO) << "=== Chunk Header ===";
      LOG(INFO) << "chunk_type: " << chunk_header->chunk_type;
      LOG(INFO) << "chunk_sz: " << chunk_header->chunk_sz;
      LOG(INFO) << "total_sz: " << chunk_header->total_sz;
    }
    if (sparse_header->chunk_hdr_sz > sizeof(chunk_header_t)) {
      /*
      * Skip the remaining bytes in a header that is longer
      * than we expected.
      */
      data += (sparse_header->chunk_hdr_sz -
        sizeof(chunk_header_t));
    }
    chunk_data_sz = sparse_header->blk_sz * chunk_header->chunk_sz;
    blkcnt = chunk_data_sz / block_size;
    switch (chunk_header->chunk_type) {
    case CHUNK_TYPE_RAW:
      if (chunk_header->total_sz !=
        (sparse_header->chunk_hdr_sz + chunk_data_sz)) {
        LOG(ERROR) << "Bogus chunk size for chunk type Raw";
        return -1;
      }
      if (blk + blkcnt > 0 + block_count) {
        LOG(ERROR) << "Request would exceed partition size!";
        return -1;
      }
      addr_offset = blk * block_size;
      ret = lseek64(fd, offset_ + addr_offset, SEEK_SET);
      if (ret < 0) {
        LOG(ERROR) << "failed to seek file to " << addr_offset << " error=" << strerror(errno);
        return -1;
      }
      written = write(fd, data, blkcnt * block_size);
      if (written < 0) {
        LOG(ERROR) << "SparseWriter: failed to write data of len ";
        return -1;
      }
      total_bytes = total_bytes + blkcnt * block_size;
      blks = written / block_size;
      blk += blks;
      bytes_written += blkcnt * block_size;
      total_blocks += chunk_header->chunk_sz;
      data += chunk_data_sz;
      break;
    case CHUNK_TYPE_FILL:
      if (chunk_header->total_sz !=
        (sparse_header->chunk_hdr_sz + sizeof(uint32_t))) {
        LOG(ERROR) << "Bogus chunk size for chunk type FILL total_sz err " << chunk_header->total_sz << "
";
        return -1;
      }
      ret = posix_memalign (&membuf, 64, 
            ROUNDUP(
            block_size * fill_buf_num_blks,
            64));
      if (ret) {
        LOG(ERROR) << "posix_memalign:" << strerror (errno);
        return -1;
      }
      fill_buf = (uint32_t *)membuf;
      if (!fill_buf) {
        LOG(ERROR) << "Malloc failed for: CHUNK_TYPE_FILL";
        return -1;
      }
      fill_val = *(uint32_t *)data;
      data = data + sizeof(uint32_t);
      for (i = 0;
        i < (block_size * fill_buf_num_blks /
         sizeof(fill_val));
        i++)
        fill_buf[i] = fill_val;
      if (blk + blkcnt > 0 + block_count) {
        LOG(ERROR) << "Request would exceed partition size!";
        return -1;
      }
      for (i = 0; i < blkcnt;) {
        j = blkcnt - i;
        if (j > fill_buf_num_blks)
          j = fill_buf_num_blks;
        addr_offset = blk * block_size;
        ret = lseek64(fd, offset_ + addr_offset, SEEK_SET);
        if (ret < 0) {
          LOG(ERROR) << "failed to lseek file to " << addr_offset << " error=" << strerror(errno);
          return -1;
        }
        written = write(fd, fill_buf, j * block_size);
        if (written < 0) {
          LOG(ERROR) << "SparseWriter: failed to write data of len ";
          return -1;
        }
        total_bytes = total_bytes + j * block_size;
        blks = written / block_size;
        if (blks < j) {
          LOG(ERROR) << "Write failed, block";
          free(fill_buf);
          return -1;
        }
        blk += blks;
        i += j;
      }
      bytes_written += blkcnt * block_size;
      total_blocks += chunk_data_sz / sparse_header->blk_sz;
      free(fill_buf);
      break;
    case CHUNK_TYPE_DONT_CARE:
      blk += blkcnt;
      total_blocks += chunk_header->chunk_sz;
      break;
    case CHUNK_TYPE_CRC32:
      if (chunk_header->total_sz !=
        sparse_header->chunk_hdr_sz) {
        LOG(ERROR) << "Bogus chunk size for chunk type CRC32 total_sz err " << chunk_header->total_sz;
        return -1;
      }
      total_blocks += chunk_header->chunk_sz;
      data += chunk_data_sz;
      break;
    default:
      LOG(INFO) << __func__ << ": Unknown chunk type: " << chunk_header->chunk_type;
      return -1;
    }
  }
  LOG(INFO) << "Wrote "<< chunk <<"blocks, expected to write " << sparse_header->total_blks << "blocks
";
  LOG(INFO) << "........ wrote "<< bytes_written <<"bytes to " << partitionName << "
";
  LOG(INFO) << "total_bytes=" << total_bytes;
  return 0;
}

3、驗證稀疏鏡像升級（1）拷貝升級包將制作的稀疏鏡像升級包通過HDC工具推進(jìn)系統(tǒng)data目錄，并修改文件名為updater.zip，路徑如下：/data/updater/updater.zip（2）啟動升級系統(tǒng)啟動后，在命令行工具輸入如下命令啟動升級reboot updater:--update_package=/data/updater/updater.zip輸入命令后，系統(tǒng)重啟，進(jìn)入升級頁面，等待升級完成。 本文介紹了OpenHarmony系統(tǒng)中實現(xiàn)稀疏鏡像升級的方法，理解稀疏鏡像原理及稀疏鏡像還原方法可以快速在自己的系統(tǒng)中應(yīng)用稀疏鏡像升級，提高系統(tǒng)升級速度。