Introduction

Keramics provides read-only access to a collection of data formats.

This document is intended as a working document of specifications of data formats used by the Keramics project. These specifications are based on available documentation and analysis of data samples.

Note that these might differ from authorative format specifications and are works in progress.

Storage media image formats

A storage media image format is used to store data from storage media devices such as a hard disk, a floppy or optical disk like CD-ROM or DVD.

Formats

• Expert Witness Compression Format (EWF)
• Expert Witness Compression Format version 2 (EWF2)
• Mac OS sparse bundle
• Mac OS sparse image
• Parallels Disk Image (PDI)
• QEMU Copy-On-Write (QCOW)
• Universal Disk Image Format (UDIF)
• Virtual Hard Disk (VHD)
• Virtual Hard Disk version 2 (VHDX)
• VMWare Virtual Disk Format (VMDK)

Expert Witness Compression Format (EWF)

EWF is short for Expert Witness Compression Format. It is a file type used to store storage media images for digital forensic purposes. It is currently widely used in the field of computer forensics in proprietary tooling like EnCase en FTK. The original specification of the format was provided by ASR Data for the SMART application.

The EWF format was succeeded by the Expert Witness Compression Format version 2 in EnCase 7 (EWF2-Ex01 and EWF2-Lx01). EnCase 7 also uses a different version of EWF-L01 then its predecessors.

Overview

The Expert Witness Compression Format (EWF) is used to store:

• storage media images, such as hard disks, USB sticks, optical disks
• individual volumes or partitions
• “physical” RAM and process memory

EWF can store data compressed or uncompressed, in a single image in one or more segment files. Each segment file consist of a standard header, followed by multiple sections. A single section cannot span multiple files. Sections are arranged back-to-back.

Terminology

In this document when referred to the EWF format it refers to the original specification by ASR Data. The newer formats like that of EnCase are deducted from the original specification and will be referred to as the EWF-E01, because of the default file extension. Whereas the Logical File Evidence (LVF) format introduced in EnCase 5, which is also stored in the EWF format will be referred to as EWF-L01. The SMART format is viewed separately to allow for discussion if the implementation differs from the specification by ASR Data and will be referred to as the EWF-S01, because of the default file extension.

All offsets are relative to the beginning of an individual section, unless otherwise noted. EnCase allows a maximum size of a segment file to be 2000 MiB. This has to do with the size of the offset of the chunk of media data. This is a 32 bit value where the most significant bit (MSB) is used as a compression flag. Therefore the maximum offset size (31 bit) can address about 2048 MiB. In EnCase 6.7 an addition was made to the table value to provide for a base offset to allow for segment files greater than 2048 MiB.

A chunk is defined as the sector size (per default 512 bytes) multiplied by the block size, the number of sectors per chunk (block) (per default 64 sectors). The data within the EWF format is stored in little-endian. The terms block and chunk are used intermittently.

Segment file

EWF stores data in one or more segment files (or segments). Each segment file consists of:

• A file header.
• One or more sections.

File header

Each segment file starts with a file header.

EWF defines that the file header consists of 2 parts, namely:

• a signature part
• fields part

EWF, EWF-E01 and SMART (EWF-S01)

The file header, used by both the EWF-E01 and SMART (EWF-S01) formats, is 13 bytes in size and consists of:

The segment number contains a number which refers to the number of the segment file, starting with 1 for the first file.

Note this means there could only be a maximum of 65535 (0xffff) files, if it is an unsigned value.

EWF-L01

The file header, used by the EWF-L01 format, is 13 bytes in size and consists of:

The segment number contains a number which refers to the number of the segment file, starting with 1 for the first file.

Note this means there could only be a maximum of 65535 (0xffff) files, if it is an unsigned value.

Segment file extensions

The SMART (EWF-S01) and the EWF-E01 formats use a different naming convention for the segment files.

SMART (EWF-S01)

The SMART (EWF-S01) extension naming has two distinct parts.

• The first segment file has the extension ‘.s01’.
  • The next segment file has the extension ’.s02.
  • This will continue up to ‘.s99’.
• After which the next segment file has the extension ‘.saa’.
  • The next segment file has the extension ‘.sab’.
  • This will continue up to ‘.saz’.
  • The next segment file has the extension ‘.sba’.
  • This will continue up to ‘.szz’.
  • The next segment file has the extension ‘.faa’.
  • This will continue up to ‘.zzz’.
  • Not confirmed but other sources report it will even continue to the use the extensions ‘.{aa’.

Keramics supports extensions up to .zzz

EWF-E01

The EWF-E01 extension naming has two distinct parts.

• The first segment file has the extension ‘.E01’.
  • The next segment file has the extension ’.E02.
  • This will continue up to ‘.E99’.
• After which the next segment file has the extension ‘.EAA’.
  • The next segment file has the extension ‘.EAB’.
  • This will continue up to ‘.EAZ’.
  • The next segment file has the extension ‘.EBA’.
  • This will continue up to ‘.EZZ’.
  • The next segment file has the extension ‘.FAA’.
  • This will continue up to ‘.ZZZ’.
  • Not confirmed but other sources report it will even continue to the use the extensions ‘.[AA’.

Keramics supports extensions up to .ZZZ

EWF-L01

The EWF-L01 extension naming has two distinct parts.

• The first segment file has the extension ‘.L01’.
  • The next segment file has the extension ’.L02.
  • This will continue up to ‘.L99’.
• After which the next segment file has the extension ‘.LAA’.
  • The next segment file has the extension ‘.LAB’.
  • This will continue up to ‘.LAZ’.
  • The next segment file has the extension ‘.LBA’.
  • This will continue up to ‘.LZZ’.
  • The next segment file has the extension ‘.MAA’.
  • This will continue up to ‘.ZZZ’.
  • Not confirmed but other sources report it will even continue to the use the extensions ‘.[AA’.

Keramics supports extensions up to .ZZZ

Segment file set identifier GUID

Segment file sets do not have a strict unique identifier. However the volume section contains a GUID that can be used for this purpose. Where:

• linen 5 to 6 use a time and MAC address based version (1) of the GUID
• EnCase 5 to 7 and linen 6 to 7 use a random based version (4) of the GUID

Note that in linen 6 the switch from a version 1 to 4 GUID was somewhere made between version 6.01 and 6.19.

See RFC4122 for more information about the different GUID versions.

The sections

The remainder of the segment file consists of sections. Every section starts with the same data this will be referred to as the section header.

Section header

The section header consist of 76 bytes, it contains information about a specific section.

Some sections contain additional data, refer to paragraph section types for more information.

Note Expert Witness 1.35 (for Windows) does not set the section size.

Note that in EnCase 2 DOS version the padding itself does not contains 0-byte values but data, probably the memory is not filled with 0-byte values.

Section types

There are multiple section types. ASR Data - E01 Compression Format defines the following:

• Header section
• Volume section
• Table section
• Next and Done section

The following sections type were found analyzing more recent EnCase files (EWF-E01):

• Header2 section
• Disk section
• Sectors section
• Table2 section
• Data section
• Error2 section
• Session section
• Hash section
• Digest section

The following sections type were found analyzing more recent EnCase files (EWF-L01):

• Ltree section
• Ltypes section

Header2 section

The header2 section is identified in the section data type field as “header2”. Some aspects of this section are:

• Found in EWF-E01 in EnCase 4 to 7, and EWF-L01 in EnCase 5 to 7
• Found at the start of the first segment file. Not found in subsequent segment files.
• The same header2 section is found twice directly after one and other.

The additional data this section contains is the following:

The information about the acquired media consists of zlib compressed data. It contains text in UTF16 format specifying information about the acquired media. The text multiple lines separated by an end of line character(s).

The first 2 bytes of the UTF16 string are the byte order mark (BOM):

• 0xff 0xfe for UTF-16 litte-endian
• 0xfe 0xff for UTF-16 big-endian

In the next paragraphs the various variants of the header2 section are described.

EnCase 4 (EWF-E01)

In EnCase 4 (EWF-E01) the header2 information consist of 5 lines, and contains the equivalent information as the header section.

The end of line character(s) is a newline (0x0a).

Note this end of line character differs from the one used in the header section.

The 3rd and the 4th line consist of the following tab (0x09) separated values.

Also see header2 values

Note the hashing algorithm is the same as for the header section.

EnCase 5 to 7 (EWF-E01)

In EnCase 5 to 7 (EWF-E01) the header2 information consist of 17 lines, and contains:

The end of line character(s) is a newline (0x0a).

Main category

The 3rd and the 4th line consist of the following tab (0x09) separated values.

Note the actual values in this category are dependent on the version of EnCase.

Also see header2 values

Note that both the acquiry and system date and time are empty in a file created by winen.

Note that the date values in the header section (not the header2 section) are set to: “Thu Jan 1 00:00:00 1970”. Where the time is dependent on the time zone and daylight savings.

Note that in a Logicube Dossier generated header2 section an additional emtpy value in the 4th line was observed. The number of values in the 3rd and 4th can differ.

Sources category

Line 6 the srce category contains information about acquisition sources.

TODO: describe what a source is in the context of EnCase.

Line 7 consists of 2 values, namely the values are “0 1”.

The 8th line consist of the following tab (0x09) separated values.

Note that the actual values in this category are dependent on the version of EnCase.

Line 9 consists of 2 values, namely the values are “0 0”.

Line 10 contains the values defined by line 8.

Note the default values of some of these values has changed around EnCase 6.12.

If the “ha” value contains “00000000000000000000000000000000” this means the MD5 hash is not set. The same applies for the “sha” value when it contains “0000000000000000000000000000000000000000” the SHA1 has is not set.

Subjects category

Line 12 the sub category contains information about subjects.

TODO: describe what a subject is in the context of EnCase.

Line 13 consists of 2 values, namely the values are “0 1”.

The 14th line consist of the following tab (0x09) separated values.

Line 15 consists of 2 values, namely the values are “0 0”.

Line 16 contains the values defined by line 14.

Note that the default values of some of these values has changed around EnCase 6.12.

EnCase 5 to 7 (EWF-L01)

The EnCase 5 to 7 (EWF-E01) header2 section specification also applies to the EnCase 5 to 7 (EWF-L01) format. However:

• both the acquired and system date and time are not set

Header2 values

Note the restrictions were tested with EnCase 7.02.01, older versions could have a restriction of 40 characters instead of 3000 characters.

Extents header value

An extents header value consist of:

number of entries
entries that consist of: S <1> <2> <3>

Header section

The header section is identified in the section data type field as “header”. Some aspects of this section are:

• Defined in ASR Data - E01 Compression Format
• Found in EWF-E01 in EnCase 1 to 7 or linen 5 to 7 or FTK Imager, EWF-L01 in EnCase 5 to 7, and SMART (EWF-S01)
• Found at the start of the first segment file or in EnCase 4 to 7 after the header2 section in the first segment file. Typically not found in subsequent segment files with the exception of Logicube Dossier generated EWF-E01 files.

The additional data this section contains is the following:

The information about the acquired media consists of zlib compressed data. It contains text in ASCII format specifying information about the acquired media. The text multiple lines separated by an end of line character(s).

In the next paragraphs the various variants of the header section are described. In all cases the information consists of at least 4 lines:

An additional 5th line is found in FTK Imager, EnCase 1 to 7 (EWF-E01).

EWF format

Some aspects of this section are:

• ASR Data - E01 Compression Format specifies the end of line character(s) is a newline (0x0a).

According to ASR Data - E01 Compression Format the 3rd and the 4th line consist of the following tab (0x09) separated values:

Also see header values

ASR Data - E01 Compression Format states that the Expert Witness Compression uses ‘f’, fastest compression.

EnCase 1 (EWF-E01)

Some aspects of this section are:

• The header section is defined only once.
• It is the first section of the first segment file. It is not found in subsequent segment files.
• The header data itself is compressed using zlib.
• The end of line character(s) is a carriage return (0x0d) followed by a newline (0x0a).

The 3rd and the 4th line consist of the following tab (0x09) separated values“

Also see header values

SMART (EWF-S01)

Some aspects of this section are:

• The header section is defined once.
• It is the first section of the first segment file. It is not found in subsequent segment files.
• The header data is always processed by zlib, however the same compression level is used as for the chunks. This could mean compression level 0 which is no compression.

The SMART format uses the FTK Imager (EWF-E01) specification for this section. Note that this could be something FTK Imager specific.

EnCase 2 and 3 (EWF-E01)

Some aspects of this section are:

• The same header section defined twice.
• It is the first and second section of the first segment file. It is not found in subsequent segment files.
• The header data itself is compressed using zlib.
• The end of line character(s) is a carriage return (0x0d) followed by a newline (0x0a).

The 3rd and the 4th line consist of the following tab (0x09) separated values:

Also see header values

EnCase 4 to 7 (EWF-E01)

Some aspects of this section are:

• The header is defined only once.
• It resides after the header2 sections of the first segment file. It is not found in subsequent segment files.
• The header data itself is compressed using zlib.
• The end of line character(s) is a carriage return (0x0d) followed by a newline (0x0a).

The 3rd and the 4th line consist of the following tab (0x09) separated values:

Also see header values

linen 5 to 7 (EWF-E01)

Some aspects of this section are:

• The same header section defined twice.
• It is the first and second section of the first segment file. It is not found in subsequent segment files.
• The header data itself is compressed using zlib.
• The end of line character(s) is a newline (0x0a).

The header information consist of 18 lines

The remainder of the string contains the following information:

The end of line character(s) is a newline (0x0a).

Main category - linen 5

The 3rd and the 4th line consist of the following tab (0x09) separated values.

Note the actual values in this category are dependent on the version of linen.

Also see header values

Main category - linen 6 to 7

The 3rd and the 4th line consist of the following tab (0x09) separated values.

Note the actual values in this category are dependent on the version of linen.

Note as of linen 6.19 the acquire date and time is in UTC and the system date and time is in local time. Where as before both values were in local time.

Also see header values

Sources category

Line 6 the srce category contains information about acquisition sources

TODO: describe what a source is in the context of EnCase.

Line 7 consists of 2 values, namely the values are “0 1”.

The 8th line consist of the following tab (0x09) separated values.

Line 9 consists of 2 values, namely the values are “0 0”.

Line 10 contains the values defined by line 8.

Note the default values of some of these values has changed around linen 6.19 or earlier.

Subjects category

Line 12 the sub category contains information about subjects.

TODO: describe what a subject is in the context of EnCase.

Line 13 consists of 2 values, namely the values are “0 1”.

The 14th line consist of the following tab (0x09) separated values.

Line 15 consists of 2 values, namely the values are “0 0”.

Line 16 contains the values defined by line 14.

Note the default values of some of these values has changed around linen 6.19 or earlier.

FTK Imager (EWF-E01)

Some aspects of this section are:

• In FTK Imager (EWF-E01) the same header section defined twice.
• It is the first and second section of the first segment file. It is not found in subsequent segment files.
• The header data itself is compressed using zlib. Note that the compression level can be none and therefore the header looks uncompressed.
• In FTK Imager the end of line character(s) is a newline (0x0a).

The 3rd and the 4th line consist of the following tab (0x09) separated values:

Also see header values

EnCase 5 to 7 (EWF-L01)

The EnCase 4 to 7 (EWF-E01) header section specification is also used for the EnCase 5 to 7 (EWF-L01) format, with the following aspects:

• In EnCase 5 both the acquired and system date and time are set to 0.
• In EnCase 6 and 7 both the acquired and system date and time are set to Jan 1, 1970 00:00:00 (the time is dependent on the local timezone and daylight savings)

Header values

Note the restrictions were tested with EnCase 7.02.01, older versions could have a restriction of 40 characters instead of 3000 characters.

Date and time header value

In EnCase a date and time contains a string of individual values separated by a space, e.g. “2002 3 4 10 19 59”, which represents March 4, 2002 10:19:59.

In linen a date and time contains a string with a POSIX 32-bit epoch timestamp, e.g. “1142163845” which represents the date: March 12 2006, 11:44:05

Extents header value

An extents header value consist of:

number of entries
entries that consist of: S <1> <2> <3>

Compression header value

A compression header value consist of a single character that represent the compression level.

Notes

There should not be a tab, carriage return and newline characters within the text in the 4th line. Or is there a method to escape these characters?

ASR Data - E01 Compression Format states that these characters should not be used in the free form text. Need to confirm this, the specification only speaks of a newline character.

Currently the password has no a additional value than allow an application check it. The data itself is not protected using the password. The password hashing algorithm is unknown. Need to find out. And does the algorithm differ per EnCase version? probably not. The algorithm does not differ in EnCase 1 to 7. FTK Imager does not bother with a password.

Volume section

The volume section is identified in the section data type field as “volume”. Some aspects of this section are:

• Defined in ASR Data - E01 Compression Format
• Found in EWF-E01 in EnCase 1 to 7 or linen 5 to 7 or FTK Imager, EWF-L01 in EnCase 5 to 7, and SMART (EWF-S01)
• Found after the header section of the first segment file. Not found in subsequent segment files.

In the next paragraphs the various versions of the volume section are described.

EWF specification

The specification according to ASR Data - E01 Compression Format.

The volume section data is 94 bytes in size and consists of:

The number of chunks is a 32-bit value this means it maximum of addressable chunks would be: 4294967295 (= 2^32 - 1). For a chunk size of 32768 x 4294967295 = about 127 TiB. The maximum segment file amount is 2^16 - 1 = 65535. This allows for an equal number of storage if a segment file is filled to its maximum number of chunks.

However Keramics is restricted at 14295 segment files, due to the extension naming schema of the segment files.

SMART (EWF-S01)

The SMART format uses the EWF specification for this section.

In SMART the signature (reverse) value is the string “SMART” (0x53 0x4d 0x41 0x52 0x54) instead of the file header signature.

FTK Imager, EnCase 1 to 7 and linen 5 to 7 (EWF-E01)

The specification for FTK Imager, EnCase 1 to 7 and linen 5 to 7.

The volume section data is 1052 bytes in size and consists of:

TODO: a value that could be in the volume is the RAID stripe size

Note that EnCase requires for media that contains no partition table that the is physical media flag is not set and vice versa. Other tools like FTK check the actual storage media data.

EnCase 5 to 7 (EWF-L01)

The EWF-L01 format uses the EnCase 5 (EWF-E01) volume section specification. However:

• the volume type contains 0x0e
• the number of chunks is 0
• the number of bytes per sectors is some kind of block size value (4096), perhaps the source file system block size
• the sectors count, represents some other value because (sector_size x sector_amount != total_size). The total size is in the ltree section.

Media type

Note that FTK imager versions, before version 2.9, set the storage media to fixed (0x01). The exact version of FTK imager where this behavior changed is unknown.

Media flags

Note that if both the the Fastbloc and Tableau write blocker media flags are set EnCase only shows the Fastbloc.

Compression level

Note that EnCase 7 no longer provides the fast and best compression options.

Disk section

The disk section is identified in the section data type field as “disk”. Some aspects of this section are:

• Not defined in ASR Data - E01 Compression Format.
• Not found in SMART (EWF-S01).

With a disk section in an FTK Imager 2.3 (EWF-E01) image it was confirmed that the disk section is the same as the volume section.

Note that the disk section was found only in FTK Imager 2.3 when acquiring a physical disk not a floppy. This requires additional research, it is currently assumed that the disk section some old method to differentiate between a partition (volume) image or a physical disk image.

Data section

The data section is identified in the section data type field as “data”. Some aspects of this section are:

• Not defined in ASR Data - E01 Compression Format.
• Found in EWF-E01 in EnCase 1 to 7 or linen 5 to 7 or FTK Imager, and EWF-L01 in EnCase 5 to 7. Not found in SMART (EWF-S01).
• For multiple segment files it does not reside in the first segment file. For a single segment file it does.
• Found after the last table2 section in a single segment file or for multiple segment files at the start of the segment files, except for the first.
• The data section has data it should should contain the same information as the volume section.

The data section is a copy of the volume section.

FTK Imager, EnCase 1 to 7 and linen 5 to 7 (EWF-E01)

Note that in Logicube products (Talon (firmware predating April 2013) and Forensic dossier (before version 3.3.3RC16)) the checksum is not calculated and set to 0.

Sectors section

The sectors section is identified in the section data type field as “sectors”. Some aspects of this section are:

• Not defined in ASR Data - E01 Compression Format.
• Found in EWF-E01 in EnCase 2 to 7, or linen 5 to 7 or FTK Imager, EWF-L01 in EnCase 5 to 7. Not found in EnCase 1 (EWF-E01) or SMART (EWF-S01).
• The first sectors section can be found after the volume section in the first segment file or at the after the data section in subsequent segment files. Successive sector data sections are found after the sector table2 section.

The sectors section contains the actual chunks of media data.

• The sectors section can contain multiple chunks.
• The default size of a chunk is 32768 bytes of data (64 standard sectors, with a size of 512 bytes per sector). It is possible in EnCase 5 and 6 and linen 5 and 6 to change the number of sectors per block to 64, 128, 256, 1024, 2048, 4096, 8192, 16384 or 32768. In EnCase 7 and linen 7 this has been reduced to 64, 128, 256, 1024.

Data chunk

The first chunk is often located directly after the section header, although the format does not require this.

When the data is compressed and the compressed data (with checksum) is larger than the uncompressed data (without the checksum) the data chunk is stored uncompressed. The default size of a chunk is 32768 bytes of data (64 standard sectors).

An uncompressed data chunk is of variable size and consists of:

The compressed data chunk consist of zlib compressed data. The checksum of the compressed data chunk is part the zlib compressed data format.

Optical disc images

For a MODE-1 CD-ROM optical disc image EnCase only seems to support 2048 bytes per sector (the data).

The raw sector size of a MODE-1 CD-ROM is 2352 bytes in size and consists of:

TODO: add information about Mode-2 and Mode-XA

Table section

The table section is identified in the section data type field as “table”. Some aspects of this section are:

• Defined in ASR Data - E01 Compression Format.
• Found in EWF-E01 in EnCase 1 to 7 or linen 5 to 7 or FTK Imager, EWF-L01 in EnCase 5 to 7, and SMART (EWF-S01)

Note that the offsets within the section header are 8 bytes (64 bits) of size while the offsets in the table entry array are 4 bytes (32 bits) in size.

In the next paragraphs the various versions of the table section are described.

EWF specification

Some aspects of the table section according to the EWF specification are:

• The first table section resides after the volume section in the first segment file or after the file header in subsequent segment files.
• It can be found in every segment file.

The table section consists of:

• the table header
• an array of table entries
• the data chunks

Table header

The table header is 24 bytes in size and consists of:

According to ASR Data - E01 Compression Format

• the number of entries, contains 0x01
• the table can hold 16375 entries if more entries are required an additional table section should be created.

Table entry

The table entry is 4 bytes in size and consists of:

The most significant bit (MSB) in the chunk data offset indicates if the chunk is compressed (1) or uncompressed (0).

A chunk data offset points to the start of the chunk of media data, which resides in the same table section within the segment file. The offset contains a value relative to the start of the file.

Data chunk

The first chunk is often located directly after the last table entry, although the format does not require this.

A data chunk is always compressed even when no compression is required. This approach provides a checksum for each chunk. The default size of a chunk is 32768 bytes of data (64 standard sectors). The resulting size of the “compressed” chunk can therefore be larger than the default chunk size.

Note that this was deducted from the behavior of FTK Imager for SMART (EWF-S01).

The compressed data chunk consist of zlib compressed data. The checksum of the compressed data chunk is part the zlib compressed data format.

SMART (EWF-S01)

The table section in the SMART (EWF-S01) format is equivalent to that of the EWF specification.

EnCase 1 (EWF-E01)

Some aspects of this section are:

• The table section resides after the volume section in the first segment file or after the file header in subsequent segment files.
• It can be found in every segment file.

The table section consists of:

• the table header
• an array of table entries
• the table footer
• the data chunks

Table header

The table header is 24 bytes in size and consists of:

The table can hold 16375 entries if more entries are required an additional table section should be created.

Table entry

The table entry is 4 bytes in size and consists of:

The most significant bit (MSB) in the chunk data offset indicates if the chunk is compressed (1) or uncompressed (0).

A chunk data offset points to the start of the chunk of media data, which resides in the same table section within the segment file. The offset contains a value relative to the start of the file.

Table footer

The table footer is 4 bytes in size and consists of:

Data chunk

The first chunk is often located directly after the table footer, although the format does not require this.

When the data is compressed and the compressed data (with checksum) is larger than the uncompressed data (without the checksum) the data chunk is stored uncompressed. The default size of a chunk is 32768 bytes of data (64 standard sectors).

An uncompressed data chunk is of variable size and consists of:

The compressed data chunk consist of zlib compressed data. The checksum of the compressed data chunk is part the zlib compressed data format.

FTK Imager and EnCase 2 to 5 and linen 5 (EWF-E01)

Some aspects of this section are:

• The table section resides after the sectors section.
• It can be found in every segment file.
• The data chunks are no longer stored in this section but in the sectors section instead.
• The table2 section contains a mirror copy of the table section. In EWF-E01 it is always present.

The table section consists of:

• the table header
• an array of table entries
• the table footer

Table header

The sector table header is 24 bytes in size and consists of:

The table section can hold 16375 entries. A new table section should be created to hold more entries. Both FTK Imager and EnCase 5 can handle more than 16375, FTK 1 cannot. To contain more than 16375 chunks new sectors, table and table2 sections need to be created after the table2 section.

Table entry

The table entry is 4 bytes in size and consists of:

The most significant bit (MSB) in the chunk data offset indicates if the chunk is compressed (1) or uncompressed (0).

A chunk data offset points to the start of the chunk of media data, which resides in the preceding sectors section within the segment file. The offset contains a value relative to the start of the file.

Table footer

The table footer is 4 bytes in size and consists of:

EnCase 6 to 7 and linen 6 to 7 (EWF-E01)

Some aspects of this section are:

• Every segment file contains its own table section.
• It resides after the sectors section.
• The data chunks are no longer stored in this section but in the sectors section instead.
• The table2 section contains a mirror copy of the table section. In EWF-E01 it is always present.

The table section consists of:

• the table header
• an array of table entries
• the table footer

Table header

The sector table header is 24 bytes in size and consists of:

As of EnCase 6 the number of entries is no longer restricted to 16375 entries. The new limit seems to be 65534.

Table entry

The table entry is 4 bytes in size and consists of:

The most significant bit (MSB) in the chunk data offset indicates if the chunk is compressed (1) or uncompressed (0).

A chunk data offset points to the start of the chunk of media data, which resides in the preceding sectors section within the segment file. The offset contains a value relative to the table base offset.

In EnCase 6.7.1 the sectors section can be larger than 2048Mb. The table entries offsets are 31 bit values in EnCase6 the offset in a table entry value will actually use the full 32 bit if the 2048Mb has been exceeded. This behavior is no longer present in EnCase 6.8 so it is assumed to be a bug. Libewf currently assumes that the if the 31 bit value overflows the following chunks are uncompressed. This allows EnCase 6.7.1 faulty EWF files to be converted.

Table footer

The table footer is 4 bytes in size and consists of:

EnCase 6 to 7 (EWF-L01)

The EWF-L01 format uses the EnCase 6 to 7 (EWF-E01) table section specification.

Table2 section

The table2 section is identified in the section data type field as “table2”. Some aspects of this section are:

• Not defined in ASR Data - E01 Compression Format.
• Found in EWF-E01 in EnCase 2 to 7, or linen 5 to 7 or FTK Imager, EWF-L01 in EnCase 5 to 7. Not found in EnCase 1 (EWF-E01) or SMART (EWF-S01).
• Uses the same format as the table section.
• Resides directly after the table section.

FTK Imager and EnCase 2 to 7 and linen 5 to 7 (EWF-E01)

The table2 section contains a mirror copy of the table section. Probably intended for recovery purposes.

EnCase 5 to 7 (EWF-L01)

The EWF-L01 format uses the EWF-E01 table2 section specification.

Next section

The next section is identified in the section data type field as “next”. Some aspects of this section are:

• Defined in ASR Data - E01 Compression Format.
• Found in EWF-E01 in EnCase 1 to 7 or linen 5 to 7 or FTK Imager, EWF-L01 in EnCase 5 to 7, and SMART (EWF-S01)
• The last section within a segment other than the last segment file.
• The offset to the next section in the section header of the next section point to itself (the start of the next section).
• It should be the last section in a segment file, other than the last segment file.

SMART (EWF-S01)

It resides after the table or table2 section.

FTK Imager, EnCase and linen (EWF-E01)

It resides after the data section in a single segment file or for multiple segment files after the table2 section.

In the EnCase (EWF-E01) format the size in the section header is 0 instead of 76 (the size of the section header).

Note that FTK imager versions before 2.9 sets the section size to 76. At the moment it is unknown in which version this behavior was changed.

Ltypes section

The ltypes section is identifier in the section data type field as “ltypes”. Some aspects of this section are:

• Found in EWF-L01 in of EnCase 7
• Found in the last segment file after table2 section before tree section.

The additional ltypes section data is 6 bytes in size and consists of:

Ltree section

The ltree section is identifier in the section data type field as “ltree”. Some aspects of this section are:

• Found in EWF-L01 in of EnCase 5 to 7
• Found in the last segment file after ltypes section and before data section.

The ltree section consists of:

• ltree header
• ltree data

Ltree header

The ltree header is 48 bytes in size and consists of:

Ltree data

The ltree data string consists of an UTF-16 little-endian encoded string without byte order mark. The ltree data is not strict UTF-16 since it allows for unpaired surrogates, such as “U+d800” and “U+dc00”.

Other observed characteristics where the names in the ltree deviate from the original source:

• [U+0001-U+0008] were converted to U+00ba
• [U+0009, U+000a] were stripped
• [U+000b, U+000c] were converted to U+0020
• U+000d was converted to U+0002
• U+00ba remained the same

Note that this behavior could be related to EnCase as well and might not be specific for EWF-L01.

The ltree data string contains the following information:

The end of line character(s) is a newline (0x0a).

Records category

The rec category contains information about records.

The 1st line of the category contains the string “rec”.

The 2nd line of the category contains tab (0x09) separated type indicators.

The 3rd line of the category consist of the tab (0x09) separated values.

Permissions category

The perm category contains information about file permissions.

The 1st line of the category contains the string “perm”.

The 2nd line consists of the following 2 values:

The 3rd line of the category contains tab (0x09) separated type indicators. For more information see the sections below.

The remaining lines in the category consist of:

• category root entry
  • zero or more permissions group entries
    • zero or more permission entries

Each entry consist of 2 lines:

The 1st line of the category root entry consists of the following 2 values:

The 1st line of the permission group entry consists of the following 2 values:

The 1st line of the permission entry consists of the following 2 values:

Permission type indicators

Permission types

Access mask

Access mask seen in combination with property types 0, 1 and 6

Access mask seen in combination with property type 2

[0x001200a9] [R&X] [R] [Sync]
[0x001301bf] [M] [R&X] [R] [W] [Sync]
[0x001f01ff] [FC] [M] [R&X] [R] [W] [Sync]

Sources category

The srce category contains information about acquisition sources of the file entries.

TODO: describe what an acquisition source is in the context of EnCase.

The 1st line of the category contains the string “srce”.

The 2nd line consists of 2 values.

The 3rd line of the category contains tab (0x09) separated type indicators. For more information see the sections below.

The remaining lines in the category consist of:

• category root
  • zero or more source entries

Each entry consist of 2 lines:

The 1st line of the category root entry consists of the following 2 values:

The 1st line of the source entry consists of the following 2 values:

Source type indicators

The acquisition date and time is in the form of: “1142163845”, which is a POSIX epoch timestamp and represents the date: March 12 2006, 11:44:05.

If the “ha” value contains “00000000000000000000000000000000” this means the MD5 hash is not set. The same applies for the “sha” value when it contains “0000000000000000000000000000000000000000” the SHA1 has is not set.

If the “ma” value contains “000000000000” this means the MAC address is not set.

Drive type

Subjects category

The sub category contains information about TODO

TODO: describe what a subject is in the context of EnCase.

The 1st line of the category contains the string “sub”.

The 2nd line consists of 2 values.

The 3rd line of the category contains tab (0x09) separated type indicators. For more information see the sections below.

The remaining lines in the category consist of:

• category root
  • zero or more subject entries

Each entry consist of 2 lines:

The 1st line of the category root entry consists of the following 2 values:

The 1st line of the subject entry consists of the following 2 values:

Subject type indicators

File entries category

The entry category contains information about the file entries.

The 1st line of the category contains the string “entry”.

The 2nd line consists of 2 values.

The 3rd line of the category contains tab (0x09) separated type indicators. For more information see the sections below.

The remaining lines in the category consist of:

• category root
  • zero or more file entries
    • zero or more sub file entries
      • …

Each entry consist of 2 lines:

The 1st line of the category root entry consists of the following 2 values:

The 1st line of the file entry consists of the following 2 values:

EnCase 5 and 6 (EWF-L01) file entry type indicators

The creation, access and last written date and time are in the form of: “1142163845”, which is a POSIX epoch timestamp and represents the date: March 12 2006, 11:44:05.

The “ha” value (Hash) consist of a MD5 hash string when file entries are hashed. If the “ha” value contains “00000000000000000000000000000000” this means the MD5 hash is not set.

Ltree file entries

The ltree entries of files and directories consist of entries starting with: 0 followed by the number of sub file entries.

The entries of files and directories:

EnCase 7 (EWF-L01) file entry type indicators

If the “ha” value contains “00000000000000000000000000000000” this means the MD5 hash is not set. The same applies for the “sha” value when it contains “0000000000000000000000000000000000000000” the SHA1 has is not set.

File entry name

A file entry name (“n” value):

• can contain path segment separator characters like “\” and “/”
• uses the “MIDDLE DOT” Unicode character (U+00b7) as a (NTFS) alternative data stream (ADS) name seperator

Note that a regular “MIDDLE DOT” Unicode character will be encoded in the same way so no real way to reliably tell the difference.

An empty name has been observed to be represented as “NoName”.

Short name

The short name (“snh”) value contains 2 values:

For example: “13 FILE10~1.TXT”

Original extents

TODO: add some text

1 30a555b 30a6000 12011ae00 9008d7 3f 43 1 12011ae00 30a6000 120113 30a6 9008d7 18530

Ltree file entries

The ltree entries of files and directories consist of entries starting with: 26 followed by the number of sub file entries.

The entries of files and directories:

File entry flags

If 0x00002000 or 0x02000000 are not set the file entry is of type “File”.

If the sparse data flag is set:

• the data size should be 1 and data should consist of a single byte value.
• the data size should be equal to the file size and data should be the same.

If the duplicate data offset value is not set the single byte value in the data should be used to reconstruct the file data. E.g. if the file size is 4096 and the data contains the byte value 0x00 the resulting file should consists of 4096 x 0x00 byte values.

If the duplicate data offset value is set the single byte in the data is ignored and the duplicate data offset refers to the location where the data stored.

Binary extents value

The binary extents value contains 3 values separated by a space:

Unknown Offset Size

Where:

• unknown always is 1, could this be the number of extents?
• extent data offset, relative from the start of the media data
• extent data size

The offset and size are specified in hexadecimal values.

Note that the binary extents value contains only 1 value for the first single file entry.

Extended attributes value

The extended attributes value contains base-16 encoded data, which consists of:

• Extended attributes header (stored as an extended attribute)
• One or more extended attributes

Extended attributes header

The extended attributes header is 37 bytes in size and consists of:

Extended attribute

An extended attributes is of variable size and consists of:

TODO: complete section

Note that branch nodes are presuably used to group attributes, however these are not used consistently and are not shown by EnCase 7.

Map section

Some aspects of this section are:

• Found in EWF-L01 in of EnCase 7 (First seen in EnCase 7.4.1.10)
• Found in the last segment file after data section before done section.

The map consists of:

• map string
• map entries array

Map string

The map string consists of an UTF-16 little-endian encoded string without the UTF-16 endian byte order mark.

The map string contains the following information:

Map string values

The number of map entries should match the number of file entries in the ltree.

Map entry

A map entry is 24 bytes in size and consists of:

Session section

The session section is identifier in the section data type field as “session”. Some aspects of this section are:

• Not defined in ASR Data - E01 Compression Format.
• It is not found in SMART (EWF-S01) and FTK Imager (EWF-E01).
• It is found in EnCase 5 and 6 (EWF-E01) files.
• It is only added to the last segment file for images of optical disc (CD/DVD/BD) media.
• It is found after the data section and before the error2 section.

The session section data consists of:

• The session header
• The session entries array
• The session footer

Session header

The session header is 36 byte in size and consists of:

Session entry

A session entry is 32 byte in size and consists of:

EnCase stores audio tracks as 0 byte data with a sector size of 2048.

Note that for a CD the first session sector is stored as 16, although the actual session starts at sector 0. Could this value be overloaded to indicate the size of the reserved space between the start of the session and the ISO 9660 volume descriptor.

Session flags

Session footer

The session footer is 4 byte in size and consists of:

Error2 section

The error2 section is identifier in the section data type field as “error2”. Some aspects of this section are:

• Not defined in ASR Data - E01 Compression Format.
• It is not found in SMART (EWF-S01).
• It is found in, EnCase 3 to 7 and linen 5 to 7 (EWF-E01) files.
• It is only added to the last segment file when errors were encountered while reading the input.

TODO: check FTK Imager, EnCase 1 and 2 for presence of the error2 section.

It contains the sectors that have read errors. The sector where a read error occurred are filled with zero’s during acquiry by EnCase.

The error2 section data consists of:

• The error2 header
• The error2 entries array
• The error2 footer

Error2 header

The error2 header is 520 byte in size and consists of:

Error2 entry

An error2 entry is 8 byte in size and consists of:

Error2 footer

The error2 footer is 4 byte in size and consists of:

Digest section

The digest section is identified in the section data type field as “digest”. Some aspects of this section are:

• It is found in EnCase 6 to 7 files, as of EnCase 6.12 and linen 6.12 (EWF-E01).

The digest section contains a MD5 and/or SHA1 hash of the data within the chunks.

The digest section data is 80 byte in size and consists of:

Hash section

The hash section is identified in the section data type field as “hash”. Some aspects of this section are:

• Defined in ASR Data - E01 Compression Format.
• It is found in SMART (EWF-S01) and FTK Imager, EnCase 1 to 7 and linen 5 to 7 (EWF-E01) files.
• It is not found in EnCase 5 (EWF-L01).
• The hash section is optional, it does not need to be present. If it does it resides in the last segment file before the done section.

The hash section contains a MD5 hash of the data within the chunks.

The hash section data is 36 byte in size and consists of:

Notes

Observations regarding the unknown value:

• is zero in SMART
• is zero in EnCase 3 and below
• in EnCase 4 the first 4 bytes are 0, the next 8 bytes seem random, the last 4 bytes seem fixed
• in EnCase 5 and 6 the first 8 bytes seem random, the last 8 bytes equal the file header signature
• in linen 5 the first and last set of 4 bytes seem the same, the second set of 4 bytes seem to be random, the third set of 4 bytes seem to contain a piece of the file header signature
• in linen 6 the first and third set of 4 bytes seem random, the second and last set of 4 bytes seem to be the same
• EnCase5 seems to contain a GUID of the acquired device?

Test with EnCase 4 show that:

• The value does not equal the checksum of the media data
• Does not differentiate for the same media acquired within the same program session, using different formats, but differ for different media and different program sessions

Done section

The done section is identified in the section data type field as “done”. Some aspects of this section are:

• Defined in ASR Data - E01 Compression Format.
• It is found in SMART (EWF-S01), FTK Imager, EnCase 1 to 7 and linen 5 to 7 (EWF-E01) and EnCase 5 (EWF-L01) files.
• The done section is the last section within the last segment file.
• The offset to the next section in the section header of the done section point to itself (the start of the done section).
• It should be the last section in the last segment file.

SMART (EWF-S01)

It resides after the table or table2 section.

FTK Imager, EnCase and linen (EWF-E01)

It resides after the data section in a single segment file or for multiple segment files after the table2 section.

In the EnCase (EWF-E01) format the size in the section header is 0 instead of 76 (the size of the section header).

Note that FTK imager versions before 2.9 sets the section size to 76. At the moment it is unknown in which version this behavior was changed.

Incomplete section

The incomplete section is identified in the section data type field as “incomplete”.

This section is seen rarely. It was seen in an EnCase 6.13 (EWF-E01) file as the last last section within the last segment file. The incomplete section was preceded by a hash and digest section, although later in the set of EWF files another hash and digest section were defined.

It is currently assumed that the incomplete section indicates an incomplete image created using remote imaging. The incomplete section contains data but currently there is no indication what purpose the data has.

EWF-X

EWF-X (extended) is an experimental format to enhance the EWF format. EWF-X is based on the EWF-E01 format. EWF-X does not limit the table entries to 16375. EWF-X is not the same as version 2 of EWF.

TODO: add note about the table entry limit.

Sections

Additional sections provided in the EWF-X format are:

• xheader
• xhash

Xheader

The xheader section contains zlib compressed data containing XML data containing the header values.

<?xml version="1.0" encoding="UTF-8"?>
<xheader>
    <case_number>1</case_number>
    <description>Description</description>
    <examiner_name>John D.</examiner_name>
    <evidence_number>1.1</evidence_number>
    <notes>Just a floppy in my system</notes>
    <acquiry_operating_system>Linux</acquiry_operating_system>
    <acquiry_date>Sat Jan 20 18:32:08 2007 CET</acquiry_date>
    <acquiry_software>ewfacquire</acquiry_software>
    <acquiry_software_version>20070120</acquiry_software_version>
</xheader>

Xhash

The xhash section contains zlib compressed data containing XML data containing the hash values.

<?xml version="1.0" encoding="UTF-8"?>
<xhash>
    <md5>ae1ce8f5ac079d3ee93f97fe3792bda3</md5>
    <sha1>31a58f090460b92220d724b28eeb2838a1df6184</sha1>
</xhash>

GUID

EWF-X uses a random based version of the GUID

Corruption scenarios

This chapter contains several corruption scenarios that have been encountered “in the wild”.

Corrupt uncompressed chunk

TODO: add description

Corrupt compressed chunk

TODO: add description

DEFLATE uncompressed block data with copy of uncompressed data size of 0

Seen in combination with some firmware versions of Tableau TD3 forensic imager.

In this corruption scenarion the copy of uncompressed data size value of the DEFLATE uncompressed block data is set to 0 instead of the 1s complement of the uncompressed data size.

Libewf currently does not handle this corruption scenario.

Corrupt section header

TODO: add description

reading section header from file IO pool entry: 1 at offset: 415912423
type                      : table2
next offset               : 415978027
size                      : 65604
checksum                  : 0xf35f03e0
number of offsets         : 16375
base offset               : 0x00000000
checksum                  : 0x180d0137

reading section header from file IO pool entry: 1 at offset: 415978027
type                      : sectors
next offset               : 415978027
size                      : 0
checksum                  : 0x1ad00464

Corrupt table section

TODO: add description

Scenarios:

• with and with out table 2
• corruption in number of entries
• corruption in entry data

Corrupted segment file header

TODO: add description

Partial segment file

TODO: add description

Missing segment file(s)

TODO: add description

Dual image: section size versus offset

The section headers define both the next section offset and the size of the section. If an implementation reads only one of the two to determine the next section, a dual EWF image can be crafted that consists of two separate images including hashes.

Keramics will mark such an image as corrupted.

Table entries offset overflow

In EnCase 6.7.1 the sectors section can be larger than 2048 MiB. The table entries offsets are 31 bit values in EnCase6 the offset in a table entry value will actually use the full 32 bit if the 2048 MiB has been exceeded. This behavior is no longer present in EnCase 6.8 so it is assumed to be a bug.

Libewf currently assumes that the if the 31 bit value overflows the following chunks are uncompressed. This allows EnCase 6.7.1 faulty EWF files to be converted by Keramics.

Multiple incomplete segment file set identifiers

Although rare it can occur that a set of EWF image files changes its segment file set identifier. This was seen in an image created by EnCase 6.13, presumably using remote imaging. The image contained 3 different segment file set identifiers. The first changes after an incomplete section. The second one changed without any clear indication. The corresponding data section also changed in some extent e.g. compression method and media flags, the is physical flag being dropped. The change was consistent across multiple segment files. It is unlikely that deliberate manipulation is involved. EnCase considers the image as invalid.

Although with some tweaking of the individual segment file sets could be read. In this case the data read from the segment file sets was heavily corrupted. For now Keramics does not support reading multiple segment files sets from a single image, but this might change in the future.

AD encryption

As of version 2.8 FTK Imager supports “AD encryption”. Although the output file uses the EWF extensions the file actually is a AES-256 encrypted container. The EWF can be encrypted using a pass-phrase or a certificate.

TODO: link to format definition

References

• ASR Data - E01 Compression Format
• RFC4122 - A Universally Unique Identifier (UUID) URN Namespace

Expert Witness Compression Format version 2 (EWF2)

TODO: add description

Mac OS sparse bundle (.sparsebundle) format

The Mac OS sparse bundle (.sparsebundle) format is one of the disk image formats supported natively by Mac OS.

The sparse bundle disk image was introduced in Mac OS X 10.5.

Overview

A sparse bundle consists of a directory (bundle) with the .sparsbundle suffix containing:

• “Info.bckup” file
• “Info.plist” file
• “token” file
• “bands” directory containing the band files

Characteristics

Info.plist and Info.bckup files

The Info.plist and its backup (Info.bckup) contain a XML plist.

This plist is also referred to as “Information Property List” and contains a single dictionary with the following key-value pairs.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>CFBundleInfoDictionaryVersion</key>
    <string>6.0</string>
    <key>band-size</key>
    <integer>8388608</integer>
    <key>bundle-backingstore-version</key>
    <integer>1</integer>
    <key>diskimage-bundle-type</key>
    <string>com.apple.diskimage.sparsebundle</string>
    <key>size</key>
    <integer>4194304</integer>
</dict>
</plist>

Token file

The token file is empty.

Bands directory

The bands directory contains files containing the actual data of the bands. The files are named using a hexadecimal naming scheme where “0” is the 1st band, “a” the 10th, “f” the 15th, “10” the 16th, etc.

Mac OS sparse image (.sparseimage) format

The Mac OS sparse image (.sparseimage) format is one of the disk image formats supported natively by Mac OS.

Overview

A sparse disk image consists of:

• header data
• bands data

Characteristics

The number of bytes per sector is 512.

Header data

The header data is 4096 bytes in size and consist of:

• file header
• band numbers array
• trailing data, which should be filled with 0-byte values

File header

The file header is 64 bytes in size and consist of:

Band numbers array

The band numbers array consists of:

• one or more band numbers

Band number

A band number is 4 bytes in size and consist of:

Where the corresponding media offset can be calculated as following:

media_offset = (band_number - 1) * sectors_per_band * 512

The offset of band data can be calculated as following:

band_data_offset = 4096 + (array_index * sectors_per_band * 512)

For example if the first array entry contains a band number of 4, then the band data is located at offset 4096 and the corresponding media offset is: 3 * sectors_per_band * 512.

Parallels Disk Image (PDI) format

The Parallels Disk Image format used in Parallels virtualization products as one of its image formats. It is both used the store hard disk images and snapshots.

Overview

A Parallels Disk Image consists of a directory, typically named “{NAME}.hdd” containing:

• Descriptor file (DiskDescriptor.xml) and backup (DiskDescriptor.xml.Backup)
• {NAME}.hdd file
• Storage data file ({NAME}.hdd.0.{GUID}.hds)
• {NAME}.hdd.drh

Where {NAME} is an arbitrary name and {GUID} is a unique identifier.

Disk types

The Parallels Disk Image format support multiple disk types:

Characteristics

The number of bytes per sector is 512.

Descriptor file

The DiskDescriptor.xml and its backup (DiskDescriptor.xml.Backup) contain the “Parallels_disk_image” XML element tha consists of the following values:

<?xml version='1.0' encoding='UTF-8'?>
<Parallels_disk_image Version="1.0">
    <Disk_Parameters>
        <Disk_size>134217728</Disk_size>
        <Cylinders>262144</Cylinders>
        <PhysicalSectorSize>4096</PhysicalSectorSize>
        <LogicSectorSize>512</LogicSectorSize>
        <Heads>16</Heads>
        <Sectors>32</Sectors>
        <Padding>0</Padding>
        <Encryption>
            <Engine>{00000000-0000-0000-0000-000000000000}</Engine>
            <Data></Data>
        </Encryption>
        <UID>{GUID}</UID>
        <Name>{NAME}</Name>
        <Miscellaneous>
            <CompatLevel>level2</CompatLevel>
            <Bootable>1</Bootable>
            <ChangeState>0</ChangeState>
            <SuspendState>0</SuspendState>
        </Miscellaneous>
    </Disk_Parameters>
    <StorageData>
        <Storage>
            <Start>0</Start>
            <End>134217728</End>
            <Blocksize>2048</Blocksize>
            <Image>
                <GUID>{GUID}</GUID>
                <Type>Compressed</Type>
                <File>{NAME}.hdd.0.{GUID}.hds</File>
            </Image>
            ...
        </Storage>
        ...
    </StorageData>
    <Snapshots>
        <Shot>
            <GUID>{GUID}</GUID>
            <ParentGUID>{GUID}</ParentGUID>
        </Shot>
        ...
    </Snapshots>
</Parallels_disk_image>

Disk parameters

The disk parameters are stored in the “Disk_Parameters” XML element and contains the following values.

Encryption

<Encryption>
    <Engine>{00000000-0000-0000-0000-000000000000}</Engine>
    <Data></Data>
    <Salt></Salt>
</Encryption>

Miscellaneous

<Miscellaneous>
    <CompatLevel>level2</CompatLevel>
    <Bootable>1</Bootable>
    <ChangeState>0</ChangeState>
    <SuspendState>0</SuspendState>
    <DupBlocksCnt>0</DupBlocksCnt>
    <CorruptBlocksCnt>0</CorruptBlocksCnt>
    <UnrefBlocksCnt>0</UnrefBlocksCnt>
    <OutOfDiskBlocksCnt>0</OutOfDiskBlocksCnt>
    <BatOverlapBlocksCnt>0</BatOverlapBlocksCnt>
    <BlocksCnt>0</BlocksCnt>
    <TruncatedBlocksCnt>0</TruncatedBlocksCnt>
    <ReferencedBlocksCnt>0</ReferencedBlocksCnt>
    <ShutdownState>0</ShutdownState>
    <GuestToolsVersion>17.1.1-51537</GuestToolsVersion>
</Miscellaneous>

CompatLevel

Seen: level0 and level2

Storage data

The “StorageData” XML element contains the following values.

Note that a split disks contains multiple “Storage” XML sub elements.

Storage

The “Storage” XML element contains the following values.

Image

The “Image” XML element contains the following values.

Snapshots data

The “Snapshots” XML element contains the following values.

Shot

The “Shot” XML element contains the following values.

Storage data file

Storage data file types

Raw storage data file

The raw (or plain) storage data file contains the disk image data including free space.

Sparse storage data file

The sparse storage data file contains the actual disk image data without free space.

A sparse storage data file consists of:

• file header
• block allocation table (BAT)
• data blocks

Sparse storage data file header

The sparse storage data file header is 64 bytes in size and consists of:

Block allocation table (BAT)

The block allocation table consists of 32-bit entries. An entry contains the sector number where the data block starts is set to 0 if the block is sparse or stored in the parent disk image.

For example block allocation table entry 0 corresponds to disk image offset 0. If contains a value of 0x800 the corresponding data block is stored at file offset 0x100000 (0x800 x 512).

QEMU Copy-On-Write (QCOW) image file format

The QEMU Copy-On-Write (QCOW) image file format is used by the QEMU Open Source Process Emulator to store disk images (storage media)

Overview

A QCOW image file consists of:

• the file header
  • optional file header extensions
• the level 1 table (cluster aligned)
• the reference count table (cluster aligned)
• reference count blocks
• snapshot headers (8-byte aligned on cluster boundary)
• clusters containing:
  • level 2 tables
  • storage media data

The storage media data is stored in clusters. Each cluster is a multitude of 512 bytes. The level 1 (L1) table contains level 1 reference of level 2 (L2) tables. The level 2 tables contain level 2 references of the storage media clusters.

There are multiple versions of the QCOW image file format. QCOW (version 1) and QCOW2 (version 2 and later) are sometimes considered even as separate image formats. Version 3 is considered as an extended version of QCOW2.

Characteristics

Note that this docuement assumes that character strings are stored in UTF-8

The number of bytes per sector is 512.

Encryption

The QCOW image format can encrypted the media data stored in the image format. Currently supported encryption methods are:

• AES-CBC 128-bit
• Linux Unified Key Setup (LUKS)

If no encryption is used the encryption method in the file header is set to none (0).

Note it is currently unknown if the format supports compression and encryption at the same time. It does not appear to be supported by qemu-img.

AES-CBC 128-bit

Both encryption and decryption use:

• AES-CBC with a 128-bits key decryption of sector data

The key is direct copy of the first 16 characters of a user provided (narrow character) password. If the password is smaller than 16 characters. The remaining key data is set to 0-byte values.

Note that it is currently unclear which character sets are allowed and how characters outside the 7-bit ASCII set should be handled.

The initialization vector of the AES-CBC is using media data sector number (relative to the start of the disk) in little-endian format as the first 64 bits of the 128 bit initialization vector. The remaining initialization vector data is set to 0-byte values. The first sector number is 0 and the bytes per sector are 512.

Linux Unified Key Setup (LUKS)

TODO: complete section

File header

File header – version 1

The file header - version 1 is 48 bytes in size and consist of:

The cluster block size is calculated as:

cluster_block_size = 1 << number_of_cluster_block_bits

The level 2 table size is calculated as:

level2_table_size = (1 << number_of_level2_table_bits) * 8

The level 1 table size is calculated as:

level1_table_entry_size = cluster_block_size * (1 << number_of_level2_table_bits)

level1_table_size = media_size / level1_table_entry_size
if media_size % level1_table_entry_size != 0:
    level1_table_size += 1

level1_table_size *= 8

The backing file name is set in snapshot image files and is normally stored after the file header.

File header – version 2

The file header - version 2 is 72 bytes in size and consist of:

The cluster block size is calculated as:

cluster_block_size = 1 << number_of_cluster_block_bits

The number of level 2 table bits is calculated as:

number_of_level2_table_bits = number_of_cluster_block_bits - 3

The level 2 table size is calculated as:

level_table2_size = (1 << number_of_level2_table_bits) * 8

The level 1 table size is calculated as:

level1_table_size = number_of_level1_table_references * 8

The backing file name is set in snapshot image files and is normally stored after the file header.

File header – version 3

The file header - version 3 is 104 or 112 bytes in size and consist of:

The cluster block size is calculated as:

cluster_block_size = 1 << number_of_cluster_block_bits

The number of level 2 table bits is calculated as:

number_of_level2_table_bits = number_of_cluster_block_bits - 3

The level 2 table size is calculated as:

level_table2_size = (1 << number_of_level2_table_bits) * 8

The level 1 table size is calculated as:

level1_table_size = number_of_level1_table_references * 8

The backing file name is set in snapshot image files and is normally stored after the file header.

Encryption methods

Incompatible feature flags

Compatible feature flags

Auto-clear feature flags

Compression methods

File header extensions

A file header extension consist of:

• file header extension header
• file header extension data

File header extension header

The file header extension header is 8 bytes in size and consist of:

File header extension types

Backing format file header extension

The backing format file header extension header is of variable size and consist of:

Bitmaps file header extension

TODO: complete section

Crypto header file header extension

The crypto header file header extension header is 16 bytes in size and consist of:

Data-file file header extension

The data-file file header extension header is of variable size and consist of:

Feature table file header extension

TODO: complete section

Level 1 table

The level 1 table contains level 2 table references.

A reference value of 0 represents unused or unallocated and is considered as sparse or stored in a corresponding backing file.

Level 2 table reference – version 1

The level 2 table reference is 8-bytes in size and consists of:

Level 2 table reference – version 2 or 3

The level 2 table reference is 8-bytes in size and consists of:

The is copied flag indicates that the reference count of the corresponding level 2 table is exactly one.

Level 2 table

The level 2 table contains cluster block references.

The level 2 table size is calculated as:

level2_table_size = (1 << number_of_level2_table_bits) * 8

A reference value of 0 represents unused or unallocated and is considered as sparse or stored in a corresponding backing file.

Cluster block reference – version 1

The cluster block reference - version 1 is 8-bytes in size and consists of:

Cluster block reference – version 2 or 3

The cluster block reference - version 2 or 3 is 8-bytes in size and consists of:

The is copied flag indicates that the reference count of the corresponding cluster block is exactly one.

Reference count table

The cluster data blocks are referenced counted. For every cluster data block a 16-bit reference count is stored in the reference count table.

The reference count table is stored in cluster block sizes. The file header contains the number of blocks (or reference count table clusters).

TODO: complete section

Notes

reference count cluster block offset = cluster data block offset /
reference count table offset = cluster data block /

In order to obtain the reference count of a given cluster, you split the
cluster offset into a refcount table offset and refcount block offset.

Since a refcount block is a single cluster of 2 byte entries, the lower
cluster_size - 1 bits is used as the block offset and the rest of the bits are
used as the table offset.

One optimization is that if any cluster pointed to by an L1 or L2 table entry
has a refcount exactly equal to one, the most significant bit of the L1/L2
entry is set as a "copied" flag. This indicates that no snapshots are using
this cluster and it can be immediately written to without having to make a copy
for any snapshots referencing it.

Cluster data block

To retrieve a cluster data block corresponding a certain storage media offset:

Determine the level 1 table index from the offset:

level1_table_index_bit_shift = number_of_cluster_block_bits + number_of_level2_table_bits

For version 1:

level1_table_index = (offset & 0x7fffffffffffffff) >> level1_table_index_bit_shift

For version 2 and 3:

level1_table_index = (offset & 0x3fffffffffffffff) >> level1_table_index_bit_shift

Retrieve the level 2 table offset from the level 1 table. If the level 2 table offset is 0 and the image has a backing file the cluster data block is stored in the backing file otherwise the cluster block is considered sparse.

Read the corresponding level 2 table.

Determine the level 2 table index from the offset:

level2_table_index_bit_mask = ~(0xffffffffffffffff << number_of_level2_table_bits)

level2_table_index = (offset >> number_of_cluster_block_bits) >> level2_table_index_bit_mask

Retrieve the cluster block offset from the level 2 table. If the cluster block offset is 0 and the image has a backing file the cluster data block is stored in the backing file otherwise the cluster block is considered sparse.

Uncompressed cluster data block

If the is compressed flag (QCOW_OFLAG_COMPRESSED) is not set:

cluster_block_bit_mask = ~(0xffffffffffffffff << number_of_cluster_block_bits)

cluster_block_data_offset = (offset & cluster_block_bit_mask) + cluster_block_offset

Note that in version 2 or 3 the last cluster block in the file can be smaller than the cluster block size defined by the number of cluster block bits in the file header. This does not seem to be the case for version 1.

Compressed cluster data block

If the is compressed flag (QCOW_OFLAG_COMPRESSED) is set the cluster block data is stored using the compression method defined by the file header or DEFLATE by default.

Multiple compressed cluster data blocks are stored together in cluster block sizes. The compressed cluster data blocks are sector (512 bytes) aligned.

The compressed data uses a DEFLATE (inflate) window bits value of -12

Compressed chunk data block – version 1

compressed_size_bit_shift = 63 - number_of_cluster_block_bits

compressed_block_size = (
    (cluster_block_offset & 0x7fffffffffffffff) >> compressed_size_bit_shift)

compressed_block_offset &= ~(0xffffffffffffffff << compressed_size_bit_shift)

Compressed chunk data block – version 2 or 3

compressed_size_bit_shift = 62 - (number_of_cluster_block_bits – 8)

According to “the QCOW2 Image Format” the compressed block size is calculated as following:

compressed_block_size = (
    (((cluster_block_offset & 0x3fffffffffffffff) >> compressed_size_bit_shift) + 1) * 512)

Since the compressed block size is stored in 512 byte sectors this value does not contain the exact byte size of the compressed cluster block data. It sometimes lacks the size of the last partially filled sector and one sector should be added if possible within the bounds of the cluster blocks size and the file size.

cluster_block_offset &= ~(0xffffffffffffffff << compressed_size_bit_shift)

Snapshots

As of version 1 QCOW can use the backing file name in the file header to point to a backing file (or parent image) that contains the snapshot image where the current image only contains the modifications. Version 2 adds support to store snapshot inside the image.

Snapshot header - version 2 or 3

An in-image snapshot is created by adding a snapshot header, copying the L1 table and incrementing the reference counts of all L2 tables and data clusters referenced by the L1 table.

The snapshot header is of variable size and consists of:

TODO: complete section

References

• The QCOW Image Format, by Mark McLoughlin
• The QCOW2 Image Format, by Mark McLoughlin

Universal Disk Image Format (UDIF)

The Universal Disk Image Format (UDIF) (.dmg) is one of the disk image formats supported natively by Mac OS.

Overview

Known UDIF image types are:

UDIF images are either uncompressed or compressed.

Uncompressed image format

An uncompressed UDIF image consist of:

• data
• optional file footer

Note that an uncompressed UDIF image without file footer is equivalent to a RAW storage media image (CRawDiskImage).

Compressed image format

A compressed UDIF image consist of:

• Data fork
• Optional resource fork
• Optional XML plist
• File footer the end of the image file

Characteristics

The number of bytes per sector is 512.

File footer

The file footer (also known as resource file or metadata) is 512 bytes in size and consists of:

Note that the XML plist size can be 0, such as in an UDIF stub (UDxx) image.

Image flags

Image types

XML plist

TODO: complete section

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>resource-fork</key>
    <dict>
        <key>blkx</key>
        <array>
            <dict>
                <key>Attributes</key>
                <string>0x0050</string>
                <key>CFName</key>
                <string>Protective Master Boot Record (MBR : 0)</string>
                <key>Data</key>
                <data>
                bWlzaAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAA
                AAgIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAIAAAAgQfL6MwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAACgAAABQAAAAMAAAAAAAAAAAAAAAAAAAABAAAA
                AAAAIA0AAAAAAAAAH/////8AAAAAAAAAAAAAAAEAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAA=
                </data>
                <key>ID</key>
                <string>-1</string>
                <key>Name</key>
                <string>Protective Master Boot Record (MBR : 0)</string>
            </dict>
            ...
        </array>
        <key>plst</key>
        <array>
            <dict>
                <key>Attributes</key>
                <string>0x0050</string>
                <key>Data</key>
                <data>
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAQAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                AAAAAAAAAAAA
                </data>
                <key>ID</key>
                <string>0</string>
                <key>Name</key>
                <string></string>
            </dict>
        </array>
    </dict>
</dict>
</plist>

The XML plist contains the following key-value pairs:

XML plist resource-fork dictionary

The resource-fork dictionary contains the following key-value pairs:

XML plist blkx array entry

A blkx array entry contains the following key-value pairs:

Block table

The block table (BLKXTable) is of variable size and consists of:

• block table header
• block table entries

The block table header

The block table header is 204 bytes in size and consists of:

Block table entry

The block table entry (BLKXChunkEntry) is 40 bytes in size and consists of:

UDIF block table entry types

UDIF comment

TODO: complete section

UDIF data fork

TODO: complete section

UDIF resource fork

TODO: complete section

Notes

Is the maximum compressed chunk size 2048 sectors?

Comment seems to reference compressed data but has no size or number of sectors value.

Virtual Hard Disk (VHD) image format

The Virtual Hard Disk (VHD) format is used by Microsoft vitualization products as one of its image formats. It is both used the store hard disk images and snapshots.

Overview

There are multiple types of VHD images, namely:

• Fixed-size VHD image
• Dynamic-size (or sparse) VHD image
• Differential (or differencing) VHD image

Fixed-size hard disk image

A fixed-size VHD image consists of:

• data
• file footer

Note that a fixed-size VHD image is equivalent to a raw storage media image with an additional footer.

Dynamic-size (or sparse) hard disk image

A dynamic-size (or sparse) VHD image consists of:

• copy of file footer
• dynamic disk header
• block allocation table
• data in blocks
• file footer

Differential hard disk image

A differential (or differencing) VHD image consists of:

• copy of file footer
• dynamic disk header
• block allocation table
• data in blocks
• file footer

Characteristics

The number of bytes per sector is 512.

Undo disk image

Virtual PC has a feature to create “Undo Disks”. This undo disk feature stores a differential hard disk image in files named something similar like:

VirtualPCUndo_<name>_0_0_hhmmssMMDDYYYY.vud

Where the date and time seems to be stored in UTC and <name> represents the name of the parent image.

File footer

The file footer is 512 bytes in size and consists of:

Features

A value of 0 represents no features are enabled.

Creator application

Creator host operating system

Disk geometry

The disk geometry is 4 bytes in size and consists of:

Disk type

Dynamic disk header

The dynamic disk header is 1024 bytes in size and consists of:

The maximum number of block allocation table entries should match the maximum possible number of blocks in the disk.

Note that the parent name can also contain a full path, e.g. in .avhd files. The part segments are separated by the \ character.

Parent locator entry

The parent locator entry is 24 bytes in size and consists of:

Locator platform code

Block allocation table

The block allocation table is only used in dynamic and differential disk images.

The block allocation table consists of 32-bit entries. An entry contains the sector number where the data block starts or is set to 0xffffffff (-1) if the block is sparse or stored in the parent disk image.

if block_allocation_table_entry == 0xffffffff:
    block is sparse or stored in parent
else:
    file_offset = (block_allocation_table_entry * 512 ) + sector_bitmap_size

Unused block in a dynamic disk are sparse and should be filled with zero byte values. In a differential disk the block is stored in the parent disk image.

Data blocks

Data blocks are only used in dynamic and differential disk images.

A data block consists of:

• sector bitmap
• sector data

size_of_bitmap (in bytes) = block_size / (512 * 8)

The size of the bitmap is rounded up to the next multitude of the sector size.

Sector bitmap

In dynamic disk images the sector bitmap indicates which sectors contain data (bit set to 1) or are sparse (bit set to 0).

In differential disk images the sector bitmap indicates which sectors are stored within the image (bit set to 1) or in the parent (bit set to 0).

The bitmap is padded to a 512-byte sector boundary.

The bitmap is stored on a per-byte basis with the MSB represents the first bit in the bitmap.

References

• VHD Specifications, by Microsoft

Virtual Hard Disk version 2 (VHDX) image format

The Virtual Hard Disk version 2 (VHDX) format is used by Microsoft vitualization products as one of its image formats. It is both used the store hard disk images and snapshots.

Overview

A VHDX image file consist of:

• file header
• 2x image headers
• 2x region tables
• log or metadata journal
• block allocation table (BAT) region
• metadata region
  • metadata table
  • metadata items
• image (content) data

The elements are stored in 64 KiB (65536 bytes) aligned blocks

Characteristics

The number of bytes per sector is 512 or 4096 depending on the logical sector size.

File hader

The file header of (file type identifier) is 64 KiB (65536 bytes) in size and consists of:

Image header

The image header is 4 KiB (4096 bytes) in size and consists of:

Checksum calculation

The CRC32-C algorithm with the Castagnoli polynomial (0x1edc6f41) and initial value of 0 is used to calculate the checksum.

The checksum is calculated over the 4 KiB bytes of data of the image header, where the image header checkum value is considered to be 0 during calculation.

Region table

The region table is stored in a block of 64 KiB (65536 bytes) and consists of:

• region table header
• 0 or more region table entries
• Unknown (reserved)

TODO: determine if 0 entries is actually supported

Region table header

The region table header is 16 bytes in size and consists of:

The CRC32-C algorithm with the Castagnoli polynomial (0x1edc6f41) and initial value of 0 is used to calculate the checksum.

The checksum is calculated over the 64 KiB bytes of data of the region table where the image header checkum value is considered to be 0 during calculation.

Region table entry

The region table entry is 32 bytes in size and consists of:

Region type identifiers

Metadata region

The metadata region contains:

• metadata table
• metadata items

Metadata table

The metadata table is stored in a block of 64 KiB (65536 bytes) and consists of:

• metadata table header
• 0 or more metadata table entries
• Unknown (reserved)

TODO: determine if 0 entries is actually supported

Metadata table header

The metadata table header is 32 bytes in size and consists of:

Metadata table entry

The metdata table entry is 32 bytes in size and consists of:

TODO: describe last 8 bytes

Metadata items

Metadata item identifiers

File parameters metadata item

The file parameters metadata item is 8 bytes in size and consists of:

Logical sector size metadata item

The logical sector size metadata item is 4 bytes in size and consists of:

Parent locator metadata item

The parent locator metadata item is of variable size and consits of:

• parent locator header
• 0 or more parent locator entry
• parent locator key and value data

TODO: determine if 0 entries is actually supported

Parent locator header

The parent locator header is 20 bytes in size and consists of:

Parent locator entry

The parent locator entry is 12 bytes in size and consists of:

Parent locator key and value data

A parent locator key or value is stored as UCS-2 little-endian string without end-of-string character.

Known keys are:

Physical sector size metadata item

The physical sector size metadata item is 4 bytes in size and consists of:

Virtual disk identifier metadata item

The virtual disk identifier metadata item is 16 bytes in size and consists of:

Note that in contrast to VHD (version 1) the virtual disk identifier does not change between a differential image and its parent. The data write identifier seems to be used instead.

Virtual disk size metadata item

The virtual disk size metadata item is 8 bytes in size and consists of:

Block allocation table (BAT) region

The block allocation table (BAT) region contains the block allocation table. The entries of this table describe the location of either blocks containing image content data (or payload blocks) or blocks containing a sector bitmap.

The size of an individual sector bitmap block is 1 MiB which allows for 2^23 sectors to be represented by the bitmap.

Block allocation table (BAT) entries are grouped in chunks. The size of a chunk can be calculated as following:

number_of_entries_per_chunk = (2^23 * logical_sector_size) / block_size

The block allocation table (BAT) consists of:

• one or more chunks containing:
  • number of entries per chunk x BAT entry describing image content data
  • 1 x BAT entry describing the a sector bitmap

Unused BAT entries are filled with 0-byte values.

The block allocation table (BAT) of:

• a fixed-size image does not contain sector bitmap entries;
• a dynamic-size image does contain sector bitmap entries, although according to MS-VHDX are not used;
• a differential image does contain sector bitmap entries.

Block allocation table (BAT) entry

The block allocation table (BAT) entry is 64 bits in size and consists of:

Block states

Payload block states

Sector bitmap block states

Sector bitmap

In differential disk images the sector bitmap indicates which sectors are stored within the image (bit set to 1) or in the parent (bit set to 0).

The bitmap is stored in a 1 MiB block.

The bitmap is stored on a per-byte basis with the LSB represents the first bit in the bitmap.

Log (metadata journal)

TODO: complete section

The log serves as metadata journal is of variable size and consist of contiguous circular (ring) buffer that contains log entries.

Log entry

TODO: complete section

4 KiB (4096 bytes) in size

Log entry header

TODO: complete section

Zero descriptor

TODO: complete section

Data descriptor

TODO: complete section

Data sector

TODO: complete section

References

• MS-VHDX: Virtual Hard Disk v2 (VHDX) File Format, by Microsoft

VMware Virtual Disk (VMDK) format

The VMware Virtual Disk (VMDK) format is used by VMware virtualization products as one of its image format.

Overview

A VMDK disk image can consist of multiple files, such as:

• descriptor file
• extent data files
• raw extent data file
• VMDK sparse extent data file
• COWD sparse extent data file

Characteristics

The number of bytes per sector is 512.

Disk types

There are multiple types of VMKD images, namely:

The 2GbMaxExtentFlat (or twoGbMaxExtentFlat) disk image, which consists of:

• a descriptor file (<name>.vmdk)
• raw data extent files (<name>-f###.vmdk), where ### is contains a decimal value starting with 1.

The 2GbMaxExtentSparse (or twoGbMaxExtentSparse) disk image, which consists of:

• a descriptor file (<name>.vmdk)
• VMDK sparse data extent files (<name>-s###.vmdk), where ### is contains a decimal value starting with 1.

The monolithicFlat disk image, which consists of:

• a descriptor file (<name>.vmdk)
• raw data extent file (<name>-f001.vmdk)

The monolithicSparse disk image, which consists of:

• VMDK sparse data extent file (<name>.vmdk) also contains the descriptor file data.

The vmfs disk image, which consists of:

• a descriptor file (<name>.vmdk)
• raw data extent file (<name>-flat.vmdk)

The vmfsSparse differential disk image, which consists of:

• a descriptor file (<name>.vmdk)
• COWD sparse data extent files (<name>-delta.vmdk)

TODO: describe more disk types

Delta links

A delta link is similar to a differential image where the image contains the changes (or delta) in comparison of a parent image. According to the Virtual Disk Format 5.0 specification one delta image can chain to another delta image.

TODO: Name <name>-delta.vmdk

Descriptor file

The descriptor file is a case-insensitive text based file that contains the following information:

• optional comment and empty lines
• header
• extent descriptions
• optional change tracking file
• disk data base (DDB)

Note that the descriptor file can contains leading and trailing whitespace. Lines are separated by a line feed character (0x0a). And leading comment (starting with #) and empty lines.

Header

The header of a descriptor file looks similar to the data below.

# Disk DescriptorFile
version=1
CID=12345678
parentCID=ffffffff
createType="twoGbMaxExtentSparse"

The header consists of the following values:

TODO: confirm if a content identifier of ‘fffffffe’ (-2) represents that the long content identifier should be used

Format versions

Encodings

Note that it is currently unknown which encodings are supported, currently it is assumed that at least the Windows codepages are supported and that the default is UTF-8.

Disk types

Extent descriptions

The extent descriptions of a descriptor file looks similar to the data below.

# Extent description
RW 4192256 SPARSE "test-s001.vmdk"

# Extent description
RW 1048576 FLAT "test-f001.vmdk" 0

The extent descriptions consists of the following values:

Extent descriptor

The extent descriptor consists of the following values:

The extent offset is specified only for flat extents and corresponds to the offset in the file or device where the extent data is located. For device-backed virtual disks (physical or raw disks) the extent offset can be non-zero. For raw extent data files the extent offset should be zero.

Extent access mode

The extent access mode consists of the following values:

Extent types

The extent type consists of the following values:

Note that VMWare Player 9 has been observed to use “FLAT” for Windows devices

Change tracking file section

The change tracking file section was introduced in version 3 and looks similar to:

# Change Tracking File
changeTrackPath="test-flat.vmdk"

The change tracking file section consists of the following values:

Disk database

The disk data base of a descriptor file looks similar to the data below.

# The Disk Data Base
#DDB

ddb.virtualHWVersion = "4"
ddb.geometry.cylinders = "16383"
ddb.geometry.heads = "16"
ddb.geometry.sectors = "63"
ddb.adapterType = "ide"
ddb.toolsVersion = "0"

The disk data base consists of the following values:

VirtualBox has been observed to use a different case for “disk” in the section header:

# The disk Data Base

Virtual hardware version

Disk adapter types

The buslogic and lsilogic values are for SCSI disks and show which virtual SCSI adapter is configured for the virtual machine. The legacyESX value is for older ESX Server virtual machines when the adapter type used in creating the virtual machine is not known.

The raw extent data file

The raw extent data file contains the actual disk data. The raw extent data file can be a file or a device.

This type of extent data file is also known as “Simple” or “Flat Extent”.

The VMDK sparse extent data file

The VMDK sparse extent data file contains the actual disk data. A VMDK sparse extent data file consists of:

• file header
• optional embedded descriptor file
• optional secondary grain directory
  • optional secondary grain tables
• (primary) grain directory
  • (primary) grain tables
• grains
• optional backup file header

This type of extent data file is also known as “Hosted Sparse Extent” or “Stream-Optimized Compressed Sparse Extent” when markers are used.

Note that the actual layout can vary per file, Stream-Optimized Compressed Sparse Extent have been observed to use secondary file headers.

Changes in format version 2:

• added encrypted disk support (though this feature never seem to never have been implemented).

Changes in format version 3:

• the size of extent files is no longer limited to 2 GiB;
• added support for persistent changed block tracking (CBT).

Note that “CBT”, the changeTrackPath value in the descriptor file references a file that describes changed areas on the virtual disk.

File header

The file header is 512 bytes in size and consists of:

The end of line characters are used to detect corruption due to file transfers that alter line end characters.

According to Virtual Disk Format 5.0 specification the maximum data number of sectors (capacity) should be a multitude of the sectors per grain. Note that it has been observed that this is not always the case.

If the primary grain directory start sector is 0xffffffffffffffff (GD_AT_END) in a Stream-Optimized Compressed Sparse Extent there should be a secondary file header stored at offset -1024 relative from the end of the file (stream) that contains the correct grain directory start sector.

Flags

The flags consist of the following values:

Compression method

The compression method consist of the following values:

Markers

The markers are used in Stream-Optimized Compressed Sparse Extents. The corresponding flag must be set for markers to be present. An example of the layout of a Stream-Optimized Compressed Sparse Extent that uses markers is:

• file header
• embedded descriptor
• compressed grain markers
• grain table marker
• grain table
• grain directory marker
• grain directory
• footer marker
• secondary file header
• end-of-stream marker

The marker

The marker is 512 bytes in size and consists of: