# Chartstamp

**Repository Path**: cse-crypto/chartstamp

## Basic Information

- **Project Name**: Chartstamp
- **Description**: The implementation of ACM MM 2022 paper "ChartStamp: Robust Chart Embedding for Real-World Applications".
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 1
- **Forks**: 0
- **Created**: 2023-02-22
- **Last Updated**: 2023-02-22

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# ChartStamp
The implementation of ACM MM 2022 paper "ChartStamp: Robust Chart Embedding for Real-World Applications".

## Introduction
This repository contains the code that implements the ChartStamp described in our paper "ChartStamp: Robust Chart Embedding for Real-World Applications" published at ACM MM 2022. ChartStamp is a chart embedding method that is robust to real-world printing/displaying and photography. Depending on target image manipulations, ChartStamp uses encoder-decoder network to embed 100, 1000, or 10,000 bits into a chart image.

## Requrements
Our code is implemented and tested on TensorFlow with the following packages and versions:
- `python=3.8.5`
- `tensorflow=1.15.1`
- `bchlib=0.14.0`
- `opencv-python=4.5.1.48`
- `pillow=8.1.0`
- `numpy=1.18.5`

Run detect_decode.py, additional package is needed:
- `pytorch=1.7.1`

## Dataset
The chart dataset we collected contains 3,525 chart images from three sources : (1) ReVision chart image corpus ^[Manolis Savva, Nicholas Kong, Arti Chhajta, Fei-Fei Li, Maneesh Agrawala, and Jeffrey Heer. 2011. Revision: Automated classification, analysis and redesign of chart images. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 393–402.]; (2) ICPR 2020 CHART competition datasets^[ICPR. 2020. CHART-Infographics 2020. https://chartinfo.github.io/.]; and (3) charts extracted from a corpus of Excel files crawled from the Internet.

This dataset can be download from following link:

- https://drive.google.com/drive/folders/1w9PAZn1Yzr82JIJQp1pJ1A-166P-x0jf?usp=sharing

## Perceptual model
The training code for perceptual model is not included in this repo. The perceptual model is trained using Enet model released [here][enet], and is trained with 1,000 chart images randomly selected from the training set and labelled semi-automatically (segmented and labelled automatically and then manually examined and corrected).

[enet]: https://github.com/kwotsin/TensorFlow-ENet

The pretrained perceptual model can be download from following link: 
- https://drive.google.com/drive/folders/19VOZYvAyrG7gp0aagDR7XGryss-wcD7H?usp=sharing

## Training
Set dataset path in train.py
- `TRAIN_PATH = DIR_OF_DATASET_IMAGES`

Set perceptual model with command:
- `--perceptual_model DIR_OF_PERCEPTUAL_MODEL`

Train model
- `python train.py`



## Pretrained model
The pretrained model for embedding 100 bit, in this version ChartStamp is robust to real-world printing and photography

Download the pretrained perceptual model from following link, put 'saved_100' in base directory: 
- https://drive.google.com/drive/folders/1BRkW2xHfk002Jtd2-pG5SrvK_70YbTC5?usp=sharing

The pretrained model for embedding 1000 bit, in this version ChartStamp is robust to real-world displaying and photography

Download the pretrained perceptual model from following link, put 'saved_1000' in base directory: 
- https://drive.google.com/drive/folders/1r4mhaqOMsadu6uJcGzsWfd2uBOv2g0rd?usp=sharing


## Detector
The detector model is trained using the EAST model released [here][EAST]. The dataset used for training detector is generated by randomly placing warped ChartStamps onto larger images.

[EAST]: https://github.com/songdejia/EAST

Download the pretrained perceptual model from following link, put 'weight' in base directory: 
- https://drive.google.com/drive/folders/1lOz1NcGbUeIO4tG3Q-M9sEChOCNFY5Ry?usp=sharing

## Embedding a message
The script `encode.py` can be used to embed a message into an image or a directory of images. The default model expects a utf-8 encoded secret that is <= 7 characters (100 bit message -> 56 bits after ECC).

Example usage:

`python encode.py --model ./save_100 --image test_img --save_dir ./embed_bit_100 --secret chart!`

## Decoding a message
The script `decode.py` can be used to decode a message from embedded chart image.

Example usage:

`python decode.py --model ./save_100 --image test_embeded_img`

## Detecting and decoding
The script `detect_decode.py` can be used to detect and decode ChartStamps in an image. This is used for real-world printing/displaying and photography when ChartStamp lies on a small part of the captured picture.

Example usage:

`python detect_decode.py --model ./save_100 --image test_embeded_img --detect_model_path ./weight/100bit_checkpoint.pth`





## Citation

If you are using our code for research purpose, please cite our paper.

```
@inproceedings{chartstamp_acmmm2022,
  author    = {Jiayun Fu, Bin B. Zhu, Haidong Zhang, Yayi Zou1, Song Ge, Weiwei Cui, Yun Wang,
Dongmei Zhang, Xiaojing Ma and Hai Jin},
  title     = {ChartStamp: Robust Chart Embedding for Real-World Applications},
  booktitle = {In Proceedings of the 30th ACM International Conference on Multimedia (MM ’22), October 10–14, 2022, Lisboa, Portugal. ACM, New York, NY, USA, 9 pages.},
  publisher = {{ACM}},
  year      = {2022},
  url       = {https://doi.org/10.1145/3503161.3548286}
}
```