Fuse Copy on Write (CoW)
Table of Contents
- Introduction
- Usage
- Implementation Details
- REST Api
Introduction
This extension to the fuse dnbd3 client allows it to mount images writable. The changes to a writable mounted image will be stored in a separate file (also called Copy on Write (Cow)) on the client computer. These changes are uploaded in the background to the cow server. Once the user unmounts the images, all remaining changes will be uploaded. Then the image will be merged on the server (if set so in the startup parameters).
Usage
New Parameters
-c <path> Enables the cow functionality, the argument sets the path for the temporary .meta and .data file in which the writes are stored-C <address> sets the address of the cow server. The cow server is responsible for merging the original image with the changes from the client.-L <path> Similar to -c <path> but instead of creating a new session, it loads an existing from the given path.-m if set, the client will request a merge after the image is unmounted and all change are uploaded.
Example parameters for creating a new cow session:
./dnbd3-fuse "/home/user/VMs/mount" -f -h localhost -i imagename -c "/home/user/temp" -C "192.168.178.20:5000"
Implementation Details
Data strucure
The datastructe is split in to main parts. The actual data from the write on the image and its corresponding metadata. Its also important to distinguish between a dnbd3 block which is 4096byte and and cow block which groups 320 dnbd3 blocks together. An cow block has an cow_block_metadata_t struct which holds the corresponding meta data. The metadata is used to determine if and block has bin written on, where this block is stored in the data file, when it was last modified and when it was uploaded. But more later.
Blockmetadata
The data structure for storing metadata about blocks contains a Layer 1(L1) and a Layer 2 (L2). L1 contains pointers to the L2's.
The whole L1 array is initialized at the beginning and cannot be resized, so the size of the L1 array limits the total size of the image.
The L2's are dynamically created once needed. So at the beginning, all L1 pointer will be null. The L2's are arrays which contains 1024
cow_block_metadata_t structs.
typedef struct cow_block_metadata
{
atomic_int_least64_t offset;
atomic_uint_least64_t timeChanged;
atomic_uint_least64_t timeUploaded;
atomic_char bitfield[40];
} cow_block_metadata_t;
Each cow_block_metadata_t contains a 40 byte so 320 bit bitfield. The bitfield indicates whether the corresponding data contains data or not. For e.g. if the bitfield starts with 01.., the first 4096 contains not data and the next 4096 contain data.
So each cow_block_metadata_t stores the metadata of up to 320*4096 byte if all bits are set to 1. The offset is the offset where in the data file is the corresponding data stored. The timeChanged property contains the unix when the block was last modified, and the timeUploaded property contains the unix time when the block was last uploaded. It's 0 if the block was never uploaded.
The L2 arrays and cow_block_metadata_t are sorted to original Offsets. So the first L1 pointer, thus the first L2 array, addresses the first 1024 * 320 * 4096 Bytes (L2Size * bitfieldsize * DNBD3Blocksize) of Data and so on.
So for example, to get the cow_block_metadata_t for offset 4033085440 you would take L1[3] since
4033085440 / ( COW_L2_STORAGE_CAPACITY ) ≈ 3.005
Then you would take the fifth cow_block_metadata_t in the L2 array because of
(4033085440 mod COW_L2_STORAGE_CAPACITY) / COW_METADATA_STORAGE_CAPACITY
Where:
COW_L2_STORAGE_CAPACITY = 1024 * 320 * 4096
COW_METADATA_STORAGE_CAPACITY = 320 * 4096
Read Request
For an read request, for every 4096byte block it will be checked if the block is already local on the computer (therefore was already written to before).
If so it will be read from disk, otherwise it will be requested from the dnbd3 server. To increase performance, multiple following blocks are also local/non local like the block before will be combined to to one larger reads from disc respectively one larger request from the server.
The graph shown above is somewhat simplified for better visibility. The reads from the server happen async. So it will not be waited for the server to respond, rather the it will move on with the next blocks. As soon as the respond from the server is finished, the data will be written in
the fuse buffer. Each request to the dnbd3 server will increase the workCounter variable by one and every time a request is done it will be decreased by one. Once workCounter is 0 again, fuse_request will be returned.
Also on the local side, it has to break the loop once the end of an cow_block_metadata_t is reached, since the next data offset of the next cow_block_metadata_t is very likely not directly after it in the data file.
Write Request
For the write request, if the start or the end or the end does not align with a multiple of 4096, then the start and/or end block must be padded.
Because every 4096byte block needs complete data, since if the bit in the bitfield for that block is set, all the data will be read locally.
To pad the block, if its still in the range of the original image size, the missing bytes will be requested from the dnbd3 server. If its outside of the original image (because the image grown in size) then the missing bytes will be padded with 0.
The write request will write compute the corresponding cow_block_metadata_t from the offset. If the corresponding cow_block_metadata_t does not exist yet it will be created. The data will be written in the data file, at the offset which is stored in the cow_block_metadata_t.
Then the corresponding bit in the bitfields will be set and the timeChanged will be updated. If there is more data to write, the next cow_block_metadata_t will be computed and the steps above will be repeated.
The workCounter variable is used here again to make sure that if padding was needed it is done before the fuse request returns.
Files
If a new CoW session is started, a new meta, data and if set so in the Command line arguments a status.txt file is created.
status.txt
While the cow session is active, the file contains:
uuid: <uuid>
state: active
- The
uuid is the session uuid, which the cow server uses to identify the session.
Once the user unmounts the image, the file contains:
uid: <uuid>
state: uploading
uploaded: <number>
totalBlocks: <number>
Once all blocks are uploaded, the state will be set to done.
meta
The meta file contains the following header:
// cowfile.h
typedef struct cowfile_metadata_header
{
uint64_t magicValue; // 8byte
atomic_uint_least64_t imageSize; // 8byte
int32_t version; // 4byte
int32_t blocksize; // 4byte
uint64_t originalImageSize; // 8byte
uint64_t metaDataStart; // 8byte
int32_t bitfieldSize; // 4byte
int32_t nextL2; // 4byte
atomic_uint_least64_t metadataFileSize; // 8byte
atomic_uint_least64_t dataFileSize; // 8byte
uint64_t maxImageSize; // 8byte
uint64_t creationTime; // 8byte
char uuid[40]; // 40byte
char imageName[200]; // 200byte
} cowfile_metadata_header_t;
After this header at byte 8192 starts the l1 and then the l2 data structure mentioned above.
data
The data files contains the magicValue and at the 40 * 8 * 4096 Offset(capacity of one cowfile_metadata_header_t) starts the first block data.
magic values in the file headers
The magic values in both files are used to ensure that a suitable file is read and that the machine has the correct endianness.
//config.h
#define COW_FILE_META_MAGIC_VALUE ((uint64_t)0xEBE44D6E72F7825E) // Magic Value to recognize a Cow meta file
#define COW_FILE_DATA_MAGIC_VALUE ((uint64_t)0xEBE44D6E72F7825F) // Magic Value to recognize a Cow data file
Threads
This extension uses two new threads:
tidCowUploader
tidStatUpdater
tidCowUploader is the thread that uploads blocks to the cow server.
tidStatUpdater updates the stats in stdout or the stats files
(depending on parameters).
Locks
This extension uses a new lock cow.l2CreateLock. It is used when a new L2 array is allocated.
Config Variables
The follwoing configuration variables have been added to config.h.
//config.h
// +++++ COW +++++
#define COW_BITFIELD_SIZE 40 // NEVER CHANGE THIS OR THE WORLD WILL ALSO END!
#define COW_FILE_META_MAGIC_VALUE ((uint64_t)0xEBE44D6E72F7825E) // Magic Value to recognize a Cow meta file
#define COW_FILE_DATA_MAGIC_VALUE ((uint64_t)0xEBE44D6E72F7825F) // Magic Value to recognize a Cow data file
#define COW_MIN_UPLOAD_DELAY 60 // in seconds
#define COW_STATS_UPDATE_TIME 5 // time in seconds the cow status files gets updated (while uploading blocks)
#define COW_MAX_PARALLEL_UPLOADS 10 // maximum number of parallel uploads
#define COW_MAX_PARALLEL_BACKGROUND_UPLOADS 2 // maximum number of parallel uploads while the image is still mounted
#define COW_URL_STRING_SIZE 500 // Max string size for an url
#define COW_SHOW_UL_SPEED 1 // enable display of ul speed in cow status file
// +++++ COW API Endpoints +++++
#define COW_API_CREATE "%s/api/File/Create"
#define COW_API_UPDATE "%s/api/File/Update?guid=%s&BlockNumber=%lu"
#define COW_API_START_MERGE "%s/api/File/StartMerge"
-
COW_MIN_UPLOAD_DELAY defines the minimum time in seconds that must have elapsed since the last modification of a cow block before it is uploaded.
-
COW_STATS_UPDATE_TIME defines the update frequency in seconds of the stdout print/ stats file update. Setting this to low could impact the performance since it hast to loop over all blocks.
COW_MAX_PARALLEL_UPLOADS defines to maximal number of parallel block uploads. These number is used once the image hast been dismounted and the final blocks are uploadedCOW_MAX_PARALLEL_BACKGROUND_UPLOADS defines to maximal number of parallel block uploads. These number is used will the image is still mounted and the user is still using it.
REST Api
To transfer the data to the cow server, the following rest api is used:
/api/File/Create
POST
Responses
| Code |
Description |
| 200 |
Success |
This request is used once a new cow session is created. The returned guid is used in all later requests to identify the session.
/api/File/Update
POST
Parameters
| Name |
Located in |
Description |
Required |
Schema |
| guid |
query |
|
Yes |
string (uuid) |
| blockNumber |
query |
|
Yes |
integer |
Responses
| Code |
Description |
| 200 |
Success |
Used for uploading a block of data. The blocknumber is the absolute block number. The body contains an "application/octet-stream" where the first bytes are the bitfield directly followed by the actual blockdata.
/api/File/StartMerge
POST
Responses
| Code |
Description |
| 200 |
Success |
| Used to start the merging on the server. |
|
/api/File/GetTopModifiedBlocks
GET
Parameters
| Name |
Located in |
Description |
Required |
Schema |
| guid |
query |
|
Yes |
string (uuid) |
| amount |
query |
|
Yes |
integer |
Responses
| Code |
Description |
| 200 |
Success |
This request returns a list that contains Block Ids and the amount of times this block got uploaded, sorted by the amount of uploads. This is useful to adjusting the COW_MIN_UPLOAD_DELAY.
/api/File/Status
GET
Parameters
| Name |
Located in |
Description |
Required |
Schema |
| guid |
query |
|
Yes |
string (uuid) |
Responses
| Code |
Description |
| 200 |
Success |
Returns the SessionStatus Model, which gives information about the session.
Models
BlockStatistics
| Name |
Type |
Description |
Required |
| blockNumber |
integer |
|
Yes |
| modifications |
integer |
|
Yes |
SessionState
| Name |
Type |
Description |
Required |
| SessionState |
string |
|
|
SessionStatus
| Name |
Type |
Description |
Required |
| state |
string |
Enum: "Copying", "Active", "Merging", "Done", "Failed" |
Yes |
| imageName |
string |
|
Yes |
| originalImageVersion |
integer |
|
Yes |
| newImageVersion |
integer |
|
Yes |
| mergedBlocks |
integer |
|
Yes |
| totalBlocks |
integer |
|
Yes |
