Google cloud storage fuse

Overview

Google cloud storage fuse 내용 정리.

Basic

Cloud storage FUSE is an open source FUSE adapter that allows user to mound Cloud Storage buckets as file systems on Linux or macOS systems. It also provides a way for applications to upload and download Cloud Storage objects using standard file system semantics. Cloud Storage FUSE can be run anywhere with connectivity to Cloud Storage, including Google Compute Engine VMs or on-premises systems.

Mount

gcsfuse 를 이용하면 google cloud storage 를 파일시스템으로 mount를 시킬 수 있다. 이 때 중요한 것은 --implicit-dirs 옵션을 붙여주여야 디렉토리 구조가 보인다는 것이다.

Implicit directories

By default, there is no allowance for the implicit existence of directories. Since the usual file system operation like mkdir will do the right thing, if the user sets up a bucket's structure using only gcsfuse then user will not notice anything odd about this. If, however, user uses some other tool to set up objects in GCS(such as the storage browser in the Google Developer Console), user may notice that not all objects are visible unitil user creates leading directories for them.

For example, say that user uses some other tool to set up a single object named "foo/bar" in user's bucket, then mount the bucket with gcsfuse. The file system will initially appear empty, since there is no "foo/" object. However if the user subsequently run `mkdir foo`, user will now see a directory named "foo" containing a file named "bar".

gcsfuse supports a flag called `--implicit-dirs` that changes the behavior. When this flag is enabled, name lookup requests from the kernel use the GCS API's Objects.list operation to search for objects that would implicitly define the existence of a directory with the name in question.

The use of `--implicit-dirs` has some drawbacks

• The feature requires an additional request to GCS for each name lookup, which may have costs in terms of request budget and latency.

• With this setup, it will appear as if there is a directory called "foo" containing a file called "bar". But when the user runs `rm foo/bar`, suddenly it will appear as if the file system is completely empty. This is contrary to expectations, since the user hasn't run `rmdir foo`.

• gcsfuse sends a single Objects.list request to GCS, and treats the directory as being implicitly defined if the results are non-empty. In rare cases (notably when many objects have recently been deleted) Objects.list may return an artibrary number of empty responses with continuation token, even for a non-empty name range. In implicitly defined directory will fail to appear.

systemd setting

[Unit]
Description=gcsfuse mount
After=network.target

[Service]
User=root
Type=forking
ExecStart=/usr/bin/gcsfuse \
  --implicit-dirs \
  --key-file /etc/service_account.json \
  test-bucket-name \
  /mnt/target_directory
ExecStop=/bin/fusermount -u /mnt/media
Restart=always

[Install]
WantedBy=multi-user.target
Alias=gcsfuse.service

일반적으로

See also

• https://cloud.google.com/storage/docs/gcs-fuse - Cloud Storage FUSE
• https://github.com/GoogleCloudPlatform/gcsfuse - GoogleCloudPlatform/gcsfuse