New Technologies File System (NTFS) format

The New Technologies File System (NTFS) format is the primary file system for Microsoft Windows versions that are based on Windows NT.

Overview

An New Technologies File System (NTFS) consists of:

• boot record
• boot loader
• Master File Table (MFT)
• Mirror Master File Table (MFT)

Characteristics

Versions

Note that the format versions mentioned above are the version as used by NTFS. Another common versioning schema uses the Windows version, e.g. NTFS 5.0 is the version of NTFS used on Windows XP which is version 3.1 in schema mentioned above.

Windows does not necessarily uses the latest format version, e.g. Windows 10 (1809) has been observed to use NTFS version 1.2 for 64k cluster block size.

Terminology

Cluster

NTFS refers to it file system blocks as clusters. Note that these are not the same as the physical clusters of a harddisk. For clarity this document will refer to these as cluster blocks. In other sources they are also referred to as logical clusters.

Typically a cluster block is 8 sectors (or 8 x 512 = 4096 bytes) in size. A cluster block number is relative to the start of the boot record.

Virtual cluster

The term virtual cluster refers to cluster blocks which are relative to the start of a data stream.

Long and short (file) name

In Windows terminology the name of a file (or directory) can either be short or long. The short name is an equivalent of the file name in the (DOS) 8.3 format. The long name is actual the (full) name of the file. The term long refers to the aspect that the name is longer than the short variant. Because most documentation refer to the (full) name as the long name, for clarity sake so will this document.

Metadata files

NTFS uses the Master File Table (MFT) to store information about files and directories. The MFT entries reference the different volume and file system metadata. There are several predefined metadata files.

The following metadata files are predefined and use a fixed MFT entry number.

The following metadata files are predefined, however the MFT entry number is commonly used but not fixed.

The boot record

The boot record is stored at the start of the volume (in the $Boot metadata file) and contains:

• the file system signature
• the BIOS parameter block
• the boot loader

Boot entry point

The boot entry point often contains a jump instruction to the boot code at offset 84 followed by a no-operation, e.g.

eb52   jmp 0x52
90     nop

Number of sectors per cluster block

The number of sectors per cluster block value as used by mkntfs is defined as following:

• Values 0 to 128 represent sizes of 0 to 128 sectors.
• Values 244 to 255 represent sizes of 2^(256-n) sectors.
• Other values are unknown.

Cluster block size

The cluster block size can be determined as following:

cluster block size = bytes per sector x sectors per cluster block

Different NTFS implementations support different cluster block sizes. Known supported cluster block size:

Note that Windows 10 (1903) requires the partition containing the NTFS file system to be aligned with the cluster block size. For example for a cluster block size of 128k the partition must 128 KiB aligned. The default partition partition alignment appears to be 64 KiB.

mkntfs restricts the cluster size to:

bytes_per_sector >= cluster_block_size > 4096 * bytes_per_sector

Master File Table (MFT) offset

The Master File Table (MFT) offset can be determined as following:

mft_offset = boot_record_offset + (mft_cluster_block_number * cluster_block_size)

The lower 32-bit part of the NTFS volume serial number is the Windows API (WINAPI) volume serial number. This can be determined by comparing the output of:

fsutil fsinfo volumeinfo C:
fsutil fsinfo ntfsinfo C:

Often the total number of sectors in the boot record will be smaller than the underlying partition. A (nearly identical) backup of the boot record is stored in last sector of cluster block, that follows the last cluster block of the volume. Often this is the 512 bytes after the last sector of the volume, but not necessarily. The backup boot record is not included in the total number of sectors.

Master File Table (MFT) and index entry size

The Master File Table (MFT) entry size and index entry size are defined as following:

• Values 0 to 127 represent sizes of 0 to 127 cluster blocks.
• Values 128 to 255 represent sizes of 2^(256-n) bytes or 2^(-n) if considered as a signed byte.
• Other values are not considered valid.

BitLocker Drive Encryption (BDE)

BitLocker Drive Encryption (BDE) uses the file system signature: “-FVE-FS-”. Where FVE is an abbreviation of Full Volume Encryption.

The data structures of BDE on Windows Vista and 7 differ.

A Windows Vista BDE volume starts with:

eb 52 90 2d 46 56 45 26 46 53 2d

A Windows 7 BDE volume starts with:

eb 58 90 2d 46 56 45 26 46 53 2d

BDE is largely a stand-alone but has some integration with NTFS.

TODO: link to BDE format documentation

Volume Shadow Snapshots (VSS)

Volume Shadow Snapshots (VSS) uses the GUID 3808876b-c176-4e48-b7ae-04046e6cc752 (stored in little-endian) to identify its data.

VSS is largely a stand-alone but has some integration with NTFS.

TODO: link to VSS format documentation

Media descriptor

The boot loader

The Master File Table (MFT)

The MFT consist of an array of MFT entries. The offset of the MFT table can be found in the volume header and the size of the MFT is defined by the MFT entry of the $MFT metadata file.

Note that the MFT can consists of multiple data ranges, defined by the data runs in the $MFT metadata file.

MFT entry

Although the size of a MFT entry is defined in the volume header is commonly 1024 bytes in size and consists of:

• The MFT entry header
• The fix-up values
• An array of MFT attribute values
• Padding, which should contain 0-byte values

Note that the MFT entry can be filled entirely with 0-byte values. Seen in Windows XP for MFT entry numbers 16 - 23.

MFT entry header

The MFT entry header (FILE_RECORD_SEGMENT_HEADER) is 42 or 48 bytes in size and consists of:

“BAAD” signature

According to NTFS documentation if during chkdsk, when a multi-sector item is found where the multi-sector header does not match the values at the end of the sector, it marks the item as “BAAD” and fill it with 0-byte values except for a fix-up value at the end of the first sector of the item. The “BAAD” signature has been seen to be used on Windows NT4 and XP.

Sequence number

According to FILE_RECORD_SEGMENT_HEADER structure the sequence number is incremented each time that a file record segment is freed; it is 0 if the segment is not used.

Base record file reference

The base record file reference is used to store additional attributes for another MFT entry, e.g. for attribute lists.

MFT entry flags

The fix-up values

The fix-up values are of variable size and consists of:

On disk the last 2 bytes for each 512 byte block is replaced by the fix-up placeholder value. The original value is stored in the corresponding fix-up (original) value array entry.

Note that there can be more fix-up values than the number of 512 byte blocks in the data.

According to MULTI_SECTOR_HEADER structure the update sequence array must end before the last USHORT value in the first sector. It also states that the update sequence array size value contains the number of bytes, but based on analysis of data samples it seems to be more likely to the number of words.

In NT4 (version 1.2) the MFT entry is 42 bytes in size and the fix-up values are stored at offset 42. This is likely where the name wfixupPattern originates from.

TODO: provide examples on applying the fix-up values.

The file reference

The file reference (FILE_REFERENCE or MFT_SEGMENT_REFERENCE) is 8 bytes in size and consists of:

Note that the index value in the MFT entry is 32-bit in size.

MFT attribute

The MFT attribute consist of:

• the attribute header
• the attribute resident or non-resident data
• the attribute name
• Unknown data, likely alignment padding (4-byte alignment)
• resident attribute data or non-resident attribute data runs
• alignment padding (8-byte alignment), can contain remnant data

MFT attribute header

The MFT attribute header (ATTRIBUTE_RECORD_HEADER) is 16 bytes in size and consists of:

MFT attribute data flags

TODO: determine the meaning of compression flag in the context of resident $INDEX_ROOT. Do the data flags have a different meaning for different attributes?

Resident MFT attribute

The resident MFT attribute data is present when the non-resident flag is not set (0). The resident data is 8 bytes in size and consists of:

TODO: determine the meaning of indexed flag bits, other than the LSB

Non-resident MFT attribute

The non-resident MFT attribute data is present when the non-resident flag is set (1). The non-resident data is 48 or 56 bytes in size and consists of:

The total size of the data runs should be larger or equal to the data size.

Note that Windows will fill data beyond the valid data size with 0-byte values. The data size remains unchanged. This applies to compressed and uncompressed data. If the first VCN is zero a valid data size of 0 represents a file entirely filled with 0-byte values.

TODO: determine the meaning of a VCN of -1

For more information about compressed MFT attributes see compression.

Attribute name

The attribute name is of variable size and consists of:

Data runs

The data runs are stored in a variable size (data) runlist. This runlist consists of runlist elements.

A runlist element is of variable size and consists of:

The data run cluster block number is a singed value, where the MSB is the singed bit, e.g. if the data run cluster block contains “dbc8” it corresponds to the 64-bit value 0xffffffffffffdbc8.

The first data run offset contains the absolute cluster block number where successive data run offsets are relative to the last data run offset.

Note that the cluster block number byte size is the first nibble when reading the byte stream, but here it is represented as the upper nibble of the first byte.

The last runlist element is (0, 0), which is stored as a 0-byte value.

According to NTFS documentation the size of the runlist is rounded up to the next multitude of 4 bytes, but based on analysis of data samples it seems that the size of the trailing data can be even larger than 3 and are not always 0-byte values.

TODO: provide examples of data runs

Sparse data runs

The MFT attribute data flag (ATTRIBUTE_FLAG_SPARSE) indicates if the data stream is sparse or not, where the runlist can contain both sparse and non-sparse data runs.

A sparse data run has a cluster block number value size of 0, representing there is no offset (cluster block number). A sparse data run is filled with 0-byte values.

Compressed data streams also define sparse data runs without setting the ATTRIBUTE_FLAG_SPARSE flag.

Note that $BadClus:$Bad also defines a data run with a cluster block number value size of 0, without setting the ATTRIBUTE_FLAG_SPARSE flag.

Compresssed data runs

The MFT attribute data flags (0x00ff) indicate if the data stream is compressed or not.

Windows supports compressed data runs for NTFS file systems with a cluster block size of 4096 bytes or less.

Windows 10 supports Windows Overlay Filter (WOF) compressed data, which stores the LZXPRESS Huffman or LZX compressed data in alternate data stream named WofCompressedData and links it to the default data stream using a reparse point.

The data is stored in compression unit blocks. A compression unit typically consists of 16 cluster blocks. However the actual value is stored in the non-resident MFT attribute.

Also see compression.

The attributes

Known attribute types

The attribute types are stored in the $AttrDef metadata file.

Attribute chains

Multiple attributes can be chained to make up a single attribute data stream, e.g. the attributes:

1. $INDEX_ALLOCATION ($I30) VCN: 0
2. $INDEX_ALLOCATION ($I30) VCN: 596

The first attribute will contain the size of the data defined by all the attributes and successive attributes should have a size of 0.

It is assumed that the attributes in a chain must be continuous and defined in-order.

The standard information attribute

The standard information attribute ($STANDARD_INFORMATION) contains the basic file entry metadata. It is stored as a resident MFT attribute.

The standard information data (STANDARD_INFORMATION) is either 48 or 72 bytes in size and consists of:

Note that MFT entries have been observed without a $STANDARD_INFORMATION attribute, but with other attributes such as $FILE_NAME and an $I30 index.

Recent version of NTFS support case-sentive file names. If a directory is case-sensitive the corresponding $STANDARD_INFORMATION attribute will have a maximum number of versions of 0 and a version number of 1.

The attribute list attribute

The attribute list attribute ($ATTRIBUTE_LIST) is used to store MFT attributes outside the MFT entry, e.g. when the MFT entry is too small to store all the attributes.

The entries in the list reference the location of MFT attributes. The attribute list attribute can be stored as either a resident (for a small amount of data) or non-resident MFT attribute.

Note that MFT entry 0 also can contain an attribute list and allows to store listed attributes beyond the first data run.

The attribute list

An attribute list consists of:

• one or more attribute list entries

The attribute list entry

An attribute list entry (ATTRIBUTE_LIST_ENTRY) is of variable size and consists of:

The file name attribute

The file name attribute ($FILE_NAME) contains the basic file system information, like the parent file entry, various date and time values and name. It is stored as a resident MFT attribute.

The file name data (FILE_NAME) is of variable size and consists of:

An MFT attribute can contain multiple file name attributes, e.g. for a separate (long) name and short name.

In several cases on a Vista NTFS volume the MFT entry contained both a DOS & Windows and POSIX name space $FILE_NAME attribute. However the directory entry index ($I30) of the parent directory only contained the DOS & Windows name.

In case of a hard link the MFT entry will contain additional file name attributes with the parent file reference of each hard link.

Namespace

Note that the Windows API function CreateFile allows to create case-sensitive file names when the flag FILE_FLAG_POSIX_SEMANTICS is set.

Long to short name conversion

A short name can be determined from a long name with the following approach. In the long name:

• ignore Unicode characters beyond the first 8-bit (extended ASCII)
• ignore control characters and spaces (character < 0x20)
• ignore non-allowed characters " * + , / : ; < = > ? \
• ignore dots except the last one, which is used for the extension
• make all letters upper case

Additional observations:

• [ or ] are replaced by an underscore (_)

Make the name unique:

1. use the characters 1 to 6 add ~1 and if the long name has an extension add the a dot and its first 3 letters, e.g. “Program Files” becomes “PROGRA~1” or “ ~PLAYMOVIE.REG“ becomes “~PLAYM~1.REG”
2. if the name already exists try ~2 up to ~9, e.g. “Program Data”, in the same directory as “Program Files”, becomes “PROGRA~2”
3. if the name already exists use a 16-bit hexadecimal value for characters 3 to 6 with ~1, e.g. “x86_microsoft-windows-r..ry-editor.resources_31bf3856ad364e35_6.0.6000.16386_en-us_f89a7b0005d42fd4” in a directory with a lot of file names starting with “x86_microsoft”, becomes “X8FCA6~1.163”

TODO: determine if the behavior is dependent on a setting that can be changed with fsutil

The volume version attribute

The volume version attribute ($VOLUME_VERSION) contains volume version.

TODO: complete section. Need a pre NTFS version 3.0 volume with this attribute. $AttrDef indicates the attribute to be 8 bytes in size.

The object identifier attribute

The object identifier attribute ($OBJECT_ID) contains distributed link tracker properties. It is stored as a resident MFT attribute.

The object identifier attribute data is either 16 or 64 bytes in size and consists of:

Droid in this context refers to CDomainRelativeObjId.

The security descriptor attribute

TODO: determine if this override any value in $Secure:$SDS?

The security descriptor attribute ($SECURITY_DESCRIPTOR) contains a Windows NT security descriptor. It can be stored as either a resident (for a small amount of data) and non-resident MFT attribute.

TODO: link to security descriptor format documentation

The volume name attribute

The volume name attribute ($VOLUME_NAME) contains the volume label. It is stored as a resident MFT attribute.

The volume name attribute data is of variable size and consists of:

The volume name attribute is used in the $Volume metadata file MFT entry.

The volume information attribute

The volume information attribute ($VOLUME_INFORMATION) contains information about the volume. It is stored as a resident MFT attribute.

The volume information attribute data is 12 bytes in size and consists of:

The volume information attribute is used in the $Volume metadata file MFT entry.

Volume flags

The data stream attribute

The data stream attribute ($DATA) contains the file data. It can be stored as either a resident (for a small amount of data) and non-resident MFT attribute.

Multiple data attributes for the same data stream can be used in the attribute list to define different parts of the data stream data. The first data stream attribute will contain the size of the entire data stream data. Other data stream attributes should have a size of 0. Also see attribute chains.

The index root attribute

The index root attribute ($INDEX_ROOT) contains the root of the index tree. It is stored as a resident MFT attribute.

Also see the index and the index root.

The index allocation attribute

The index allocation attribute ($INDEX_ALLOCATION) contains an array of index entries. It is stored as a non-resident MFT attribute.

The index allocation attribute itself does not define which attribute type it contains in the index value data. For this information it needs the corresponding index root attribute.

Multiple index allocation attributes for the same index can be used in the attribute list to define different parts of the index allocation data. The first index allocation attribute will contain the size of the entire index allocation data. Other index allocation attributes should have a size of 0. Also see attribute chains.

Also see the index.

The bitmap attribute

The bitmap attribute ($BITMAP) contains the allocation bitmap. It can be stored as either a resident (for a small amount of data) and non-resident MFT attribute.

It is used to maintain information about which entry is used and which is not. Every bit in the bitmap represents an entry. The index is stored byte-wise with the LSB of the byte corresponds to the first allocation element. The allocation element can represent different things:

• an MFT entry in the MFT (nameless) bitmap;
• an index entry in an index ($I30).

The allocation element is allocated if the corresponding bit contains 1 or unallocated if 0.

The symbolic link attribute

The symbolic link attribute ($SYMBOLIC_LINK) contains a symbolic link.

TODO: complete section. Need a pre NTFS version 3.0 volume with this attribute. $AttrDef indicates the attribute is of variable size.

The reparse point attribute

The reparse point attribute ($REPARSE_POINT) contains information about a file system-level link. It is stored as a resident MFT attribute.

Als see the reparse point.

The (HPFS) extended attribute information

The (HPFS) extended attribute information ($EA_INFORMATION) contains information about the extended attribute ($EA).

The extended attribute information data is 8 bytes in size and consists of:

The (HPFS) extended attribute

The (HPFS) extended attribute ($EA) contains the extended attribute data.

The extended attribute data is of variable size and consists of:

TODO: determine if the name is 2-byte aligned

Extended attribute flags

TODO: determine what the NEED_EA flag is used for

UNITATTR extended attribute value data

The property set attribute

The property set attribute ($PROPERTY_SET) contains a property set.

TODO: complete section. Need a pre NTFS version 3.0 volume with this attribute. $AttrDef does not seem to always define this attribute.

The logged utility stream attribute

TODO: complete section

The attribute types

The attribute types are stored in the $AttrDef metadata file.

The index

The index structures are used for various purposes one of which are the directory entries.

The root of the index is stored in index root. The index root attribute defines which type of attribute is stored in the index and the root index node.

If the index is too large part of the index is stored in an index allocation attribute with the same attribute name. The index allocation attribute defines a data stream which contains index entries. Each index entry contains an index node.

An index consists of a tree, where both the branch and index leaf nodes contain the actual data. E.g. in case of a directory entries index, any node that contains index value data make up for the directory entries.

The index value data in a branch node signifies the upper bound of the values in the that specific branch. E.g. if directory entries index branch node contains the name “textfile.txt” all names in that index branch are smaller than “textfile.txt”.

Note the actual sorting order is dependent on the collation type defined in the index root attribute.

The index allocation attribute is accompanied by a bitmap attribute with the corresponding attribute name. The bitmap attribute defines the allocation of virtual cluster blocks within the index allocation attribute data stream.

Note that the index allocation attribute can be present even though it is not used.

Common used indexes

Indexes commonly used by NTFS are:

The index root

The index root consists of:

• index root header
• index node header
• an array of index values

The index root header

The index root header is 16 bytes in size and consists of:

Note that for NTFS version 1.2 the index entry size does not have to match the index entry size in the volume header. The correct size seems to be the value in the index root header.

Collation type

The index entry

The index entry consists of:

• the index entry header
• the index node header
• The fix-up values
• alignment padding (8-byte alignment), contains zero-bytes
• an array of index values

The index entry header

The index entry header is 24 bytes in size and consists of:

Note that there can be more fix-up value than supported by the index entry data size.

The index node header

The index node header is 16 bytes in size and consists of:

In an index entry (index allocation attribute) the index node size includes the size of the fix-up values and the alignment padding following it.

The remainder of the index node contains remnant data and/or zero-byte values.

The index node flags

The index value

The index value is of variable size and consists of:

The index values are stored 8 byte aligned.

Note that some other sources define the index value flags as a 16-bit value followed by 2 bytes of padding.

The index value flags

Index key and value data

Directory entry index value

The MFT attribute name of the directory entry index is: $I30.

The directory entry index value contains a file name attribute in the index key data.

Note that the index value data can contain remnant data.

The short and long names of the same file have a separate index values. The short name uses the DOS name space and the long name the WINDOWS name space. Index values with a single name use either the POSIX or DOS_WINDOWS name space.

A hard link to a file in the same directory has separate index values.

Security descriptor hash index value

The MFT attribute name of the security descriptor hash index is: $SDH. It appears to only to be used by the $Secure metadata file.

Also see the security descriptor hash index value.

Security descriptor identifier index value

The MFT attribute name of the security descriptor identifier index is: $SII. It appears to only to be used by the $Secure metadata file.

Also see the security descriptor identifier index value.

Compression

Compressed data-runs

NTFS compression groups 16 cluster blocks together. This group of 16 cluster blocks also named a compression unit, which is either “compressed” or uncompressed.

The term compressed is quoted here because the group of cluster blocks can also contain uncompressed data. A group of cluster blocks is “compressed” when it is compressed size is smaller than its uncompressed data size. Within a group of cluster blocks each of the 16 blocks is “compressed” individually.

The compression unit size is stored in the non-resident MFT attribute. The maximum uncompressed data size is always the cluster size (in most case 4096).

Note that a resident $DATA attribute with the compression type in the data flags is stored uncompressed.

The data runs in the $DATA attribute define cluster block ranges, e.g.

21 02 35 52

This data run defines 2 data blocks starting at block number 21045 followed by 14 sparse blocks. The total number of blocks in the compression unit is 16. Compressed data is stored in the first 2 blocks and the 14 sparse blocks are only there to make sure the data runs add up to the compression unit size. They do not define actual sparse data.

Another example:

21 40 37 52

This data run defines 64 data blocks starting at block number 21047. Since this data run is larger than the compression unit size the data is stored uncompressed.

If the data run was e.g. 60 data blocks followed by 4 sparse blocks the first 3 compression units (blocks 1 to 48) would be uncompressed and the last compression unit (blocks 49 to 64) would be compressed.

Also “sparse data” and “sparse compression unit” data runs can be mixed. If in the previous example the 60 data blocks would be followed by 20 sparse blocks the last compression unit (blocks 65 to 80) would be sparse.

A compression unit can consists of multiple compressed data runs, e.g. 1 data block followed by 4 data blocks followed by 11 sparse blocks. Data runs have been observed where the last data run size does not align with the compression unit size.

The sparse blocks data run can be stored in a subsequent attribute in an attribute chain and can be stored in multiple data runs.

NTFS compression stores the “compressed” data in blocks. Each block has a 2 byte block header.

The block is of variable size and consists of:

The upper 4 bits of the block size are used as flags:

TODO: link to LZNT1 documentation

Windows Overlay Filter (WOF) compressed data

A MFT entry that contains Windows Overlay Filter (WOF) compressed data has the following attributes:

• reparse point attribute with tag 0x80000017, which defines the compression method
• a nameless data attribute that is sparse and contains the uncompressed data size
• a data attribute named WofCompressedData that contains LZXPRESS Huffman or LZX compressed data

Note that if the chunk size equals the size of the uncompressed data the chunk is stored (as-is) uncompressed.

The size of the chunk offset table is:

number of chunk offsets = uncompressed size / compression unit size

The offset of the first compressed data chunk is at the end of the chunk offset table and is not stored in the chunk offset table.

If the uncompressed size of a chunk is smaller than the compression unit size the chunk is stored uncompressed.

Also see Windows Overlay Filter (WOF) compression method.

The reparse point

The reparse point is used to create file system-level links. Reparse data is stored in the reparse point attribute. The reparse point data (REPARSE_DATA_BUFFER) is of variable size and consists of:

TODO: determine if non-native (Microsoft) reparse points are stored with their GUID

The reparse point tag

Reparse point tag flags

Known reparse point tags

Junction or mount point reparse data

A reparse point with tag IO_REPARSE_TAG_MOUNT_POINT (0xa0000003) contains junction or mount point reparse data. The junction or mount point reparse data is of variable size and consists of:

Note that it is currently unclear if the names contain an end-of-string character or if they are followed by alignment padding.

TODO: determine what character values like 0x0002 represent in the substitute name

00000010: 5c 00 3f 00 3f 00 02 00  43 00 3a 00 5c 00 55 00   \.?.?... C.:.\.U.
00000020: 73 00 65 00 72 00 73 00  5c 00 74 00 65 00 73 00   s.e.r.s. \.t.e.s.
00000030: 74 00 5c 00 44 00 6f 00  63 00 75 00 6d 00 65 00   t.\.D.o. c.u.m.e.
00000040: 6e 00 74 00 73 00 00 00                            n.t.s...

Symbolic link reparse data

A reparse point with tag IO_REPARSE_TAG_SYMLINK (0xa000000c) contains symbolic link reparse data. The symbolic link reparse data is of variable size and consists of:

Symbolic link flags

Windows Overlay Filter (WOF) reparse data

A reparse point with tag IO_REPARSE_TAG_WOF (0x80000017) contains Windows Overlay Filter (WOF) reparse data. The Windows Overlay Filter (WOF) reparse data is 16 bytes in size and consists of:

Windows Overlay Filter (WOF) compression method

TODO: link to LZXPRESS Huffman and LZX documentation

Windows Container Isolation (WCI) reparse data

A reparse point with tag IO_REPARSE_TAG_WCI (0x80000018) contains Windows Container Isolation (WCI) reparse data. The Windows Container Isolation (WCI) reparse data is of variable size and consists of:

The allocation bitmap

The metadata file $Bitmap contains the allocation bitmap.

Every bit in the allocation bitmap represents a block the size of the cluster block, where the LSB is the first bit in a byte.

TODO: describe what the $SRAT data stream is used for.

Access control

The $Secure metadata file contains the security descriptors used for access control.

Security descriptor hash ($SDH) index

The security descriptor hash index value

Security descriptor identifier ($SII) index

The security descriptor identifier index value

TODO: describe the hash algorithm

Security descriptor ($SDS) data stream

TODO: link to security descriptor format documentation

The object identifiers

$ObjID:$O

Metadata transaction journal (log file)

TODO: complete section

The metadata file $LogFile contains the metadata transaction journal and consists of:

• Log File Service restart page header
• The fix-up values

Log File service restart page header

The Log File service restart page header (LFS_RESTART_PAGE_HEADER) is 30 bytes in size and consists of:

Log File service restart page versions

USN change journal

The metadata file $Extend$UsnJrnl contains the USN change journal. It is a sparse file in which NTFS stores records of changes to files and directories. Applications make use of the journal to respond to file and directory changes as they occur, like e.g. the Windows File Replication Service (FRS) and the Windows (Desktop) Search service.

The USN change journal consists of:

• the $UsnJrnl:$Max data stream, containing metadata like the maximum size of the journal
• the $UsnJrnl:$J data stream, containing the update (or change) entries. The $UsnJrnl:$J data stream is sparse.

USN change journal metadata

The USN change journal metadata is 32 bytes in size and consists of:

USN change journal entries

The $UsnJrnl:$J data stream consists of an array of USN change journal entries. The USN change journal entries are stored on a per block-basis and 8-byte aligned. Therefore the remainder of the block can contain 0-byte values.

TODO: describe journal block size

Once the stream reaches maximum size the earliest USN change journal entries are removed from the stream and replaced with a sparse data run.

USN change journal entry

The USN change journal entry (USN_RECORD_V2) is of variable size and consists of:

Update reason flags

Update source flags

Alternate data streams (ADS)

ms-properties

The ms-properties alternate data stream contains a Windows Serialized Property Store (SPS).

TODO: link to Windows Serialized Property Store (SPS) format documentation

Zone.Identifier

The Zone.Identifier alternate data stream contains ASCII text in the form:

[ZoneTransfer]
ZoneId=3

Where ZoneId refers to the Internet Explorer URL security zone of the origin.

Transactional NTFS (TxF)

As of Vista Transactional NTFS (TxF) was added.

In TxF the resource manager (RM) keeps track of transactional metadata and log files. The TxF related metadata files are stored in the metadata directory:

$Extend\$RmMetadata

Resource manager repair information

The resource manager repair information metadata file: $Extend$RmMetadata$Repair consists of the following data streams:

• the default (unnamed) data stream
• the $Config data stream, contains the resource manager repair configuration information

TODO: determine the purpose of the default (unnamed) data stream

Resource manager repair configuration information

TODO: complete section

The $Repair:$Config data streams contains:

Transactional NTFS (TxF) metadata directory

TODO: complete section

The transactional NTFS (TxF) metadata directory: $Extend$RmMetadata$Txf is used to isolate files for delete or overwrite operations.

TxF Old Page Stream (TOPS) file

The TxF Old Page Stream (TOPS) file: $Extend$RmMetadata$TxfLog$Tops consists of the following data streams:

• the default (unnamed) data stream, contains metadata about the resource manager, such as its GUID, its CLFS log policy, and the LSN at which recovery should start
• the $T data stream, contains the file data that is partially overwritten by a transaction as opposed to a full overwrite, which would move the file into the Transactional NTFS (TxF) metadata directory

TxF Old Page Stream (TOPS) metadata

TODO: complete section

The $Tops default (unnamed) data streams contains:

TxF Old Page Stream (TOPS) file data

The $Tops:$T data streams contains the file data that is partially overwritten by a transaction. It consists of multiple pending transaction XML-documents.

TODO: describe start of each sector containing 0x0001

A pending transaction XML-document starts with an UTF-8 byte-order-mark. Is roughly contains the following data:

<?xml version='1.0' encoding='utf-8'?>
<PendingTransaction Version="2.0" Identifier="...">
   <Transactions>
      <Transaction TransactionId="...">
      <Install Application="..., Culture=..., Version=..., PublicKeyToken=...,
                           ProcessorArchitecture=..., versionScope=..."
               RefGuid="..."
               RefIdentifier="..."
               RefExtra="..."/>
      ...
      </Transaction>
   </Transactions>
   <ChangeList>
      <Change Family="..., Culture=..., PublicKeyToken=...,
                     ProcessorArchitecture=..., versionScope=..."
              New="..."/>
      ...
   </ChangeList>
   <POQ>
      <BeginTransaction id="..."/>

      <CreateFile path="..."
                  fileAttribute="..."/>
      <DeleteFile path="..."/>
      <MoveFile source="..." destination="..."/>
      <HardlinkFile source="..." destination="..."/>
      <SetFileInformation path="..."
                          securityDescriptor="binary base64:..."
                          flags="..."/>

       <CreateKey path="..."/>
       <SetKeyValue path="..."
                    name="..."
                    type="..."
                    encoding="base64"
                    value="..."/>
      <DeleteKeyValue path="..."
                      name="..."/>

      ...
   </POQ>
   <InstallerQueue Length="...">
      <Action Installer="..."
              Mode="..."
              Phase="..."
              Family="..., Culture=..., PublicKeyToken=...,
                     ProcessorArchitecture=..., versionScope=..."
              Old="..."
              New="..."/>

      ...
   </InstallerQueue >
</PendingTransaction>

Transactional NTFS (TxF) Common Log File System (CLFS) files

TxF uses a Common Log File System (CLFS) log store and the logged utility stream attribute named $TXF_DATA.

TODO: link to CLFS format documentation

The base log file (BLF) of the TxF log store is:

$Extend\$RmMetadata\$TxfLog\TxfLog.blf

Commonly the corresponding container files are:

$Extend\$RmMetadata\$TxfLog\TxfLogContainer00000000000000000001
$Extend\$RmMetadata\$TxfLog\TxfLogContainer00000000000000000002

TxF uses a multiplexed log store which contains the following streams:

• the KtmLog stream used for Kernel Transaction Manager (KTM) metadata records
• TxfLog stream, which contains the TxF log records.

Transactional data logged utility stream attribute

The transactional data ($TXF_DATA) logged utility stream attribute is 56 bytes in size and consist of:

Note that a single MFT entry can contain multiple Transactional data logged utility stream attributes.

Windows definitions

File attribute flags

The file attribute flags consist of the following values:

The following flags are mainly used in the file name attribute and sparsely in the standard information attribute. It could be that they have a different meaning in both types of attributes or that the standard information flags are not updated. For now the latter is assumed.

Corruption scenarios

Data steam with inconsistent data flags

An MFT entry contains an $ATTRIBUTE_LIST attribute that contains multiple $DATA attributes. The $DATA attributes define a LZNT1 compressed data stream though only the first $DATA attribute has the compressed data flag set.

Note that it is unclear if this is a corruption scenario or not.

MFT entry: 220 information:
    Is allocated                   : true
    File reference                 : 220-59
    Base record file reference     : Not set (0)
    Journal sequence number        : 51876429013
    Number of attributes           : 5

Attribute: 1
    Type                           : $STANDARD_INFORMATION (0x00000010)
    Creation time                  : Jun 05, 2019 06:56:26.032730300 UTC
    Modification time              : Oct 05, 2019 06:56:04.150940700 UTC
    Access time                    : Oct 05, 2019 06:56:04.150940700 UTC
    Entry modification time        : Oct 05, 2019 06:56:04.150940700 UTC
    Owner identifier               : 0
    Security descriptor identifier : 5862
    Update sequence number         : 11553149976
    File attribute flags           : 0x00000820
       Should be archived (FILE_ATTRIBUTE_ARCHIVE)
       Is compressed (FILE_ATTRIBUTE_COMPRESSED)

Attribute: 2
    Type                           : $ATTRIBUTE_LIST (0x00000020)

Attribute: 3
    Type                           : $FILE_NAME (0x00000030)
    Parent file reference          : 33996-57
    Creation time                  : Jun 05, 2019 06:56:26.032730300 UTC
    Modification time              : Oct 05, 2019 06:56:03.510061800 UTC
    Access time                    : Oct 05, 2019 06:56:03.510061800 UTC
    Entry modification time        : Oct 05, 2019 06:56:03.510061800 UTC
    File attribute flags           : 0x00000020
       Should be archived (FILE_ATTRIBUTE_ARCHIVE)
    Namespace                      : POSIX (0)
    Name                           : setupapi.dev.20191005_085603.log

Attribute: 4
    Type                           : $DATA (0x00000080)
    Data VCN range                 : 513 - 1103
    Data flags                     : 0x0000

Attribute: 5
    Type                           : $DATA (0x00000080)
    Data VCN range                 : 0 - 512
    Data size                      : 4487594 bytes
    Data flags                     : 0x0001

Directory entry with outdated file reference

The directory entry: \ProgramData\McAfee\Common Framework\Task\5.ini

File entry:
    Path                           : \ProgramData\McAfee\Common Framework\Task\5.ini
    File reference                 : 51106-400
    Name                           : 5.ini
    Parent file reference          : 65804-10
    Size                           : 723
    Creation time                  : Sep 16, 2011 20:47:54.561041200 UTC
    Modification time              : Apr 07, 2012 21:07:02.684060000 UTC
    Access time                    : Apr 07, 2012 21:07:02.652810200 UTC
    Entry modification time        : Apr 07, 2012 21:07:02.684060000 UTC
    File attribute flags           : 0x00002020
       Should be archived (FILE_ATTRIBUTE_ARCHIVE)
       Content should not be indexed (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED)

The corresponding MFT entry:

MFT entry: 51106 information:
    Is allocated                   : true
    File reference                 : 51106-496
    Base record file reference     : Not set (0)
    Journal sequence number        : 0
    Number of attributes           : 3

Attribute: 1
    Type                           : $STANDARD_INFORMATION (0x00000010)
    Creation time                  : Sep 16, 2011 20:47:54.561041200 UTC
    Modification time              : Apr 07, 2012 21:07:02.684060000 UTC
    Access time                    : Apr 07, 2012 21:07:02.652810200 UTC
    Entry modification time        : Apr 07, 2012 21:07:02.684060000 UTC
    Owner identifier               : 0
    Security descriptor identifier : 1368
    Update sequence number         : 1947271600
    File attribute flags           : 0x00002020
       Should be archived (FILE_ATTRIBUTE_ARCHIVE)
       Content should not be indexed (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED)

Attribute: 2
    Type                           : $FILE_NAME (0x00000030)
    Parent file reference          : 65804-10
    Creation time                  : Sep 16, 2011 20:47:54.561041200 UTC
    Modification time              : Apr 07, 2012 21:07:02.652810200 UTC
    Access time                    : Apr 07, 2012 21:07:02.652810200 UTC
    Entry modification time        : Apr 07, 2012 21:07:02.652810200 UTC
    File attribute flags           : 0x00002020
       Should be archived (FILE_ATTRIBUTE_ARCHIVE)
       Content should not be indexed (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED)
    Namespace                      : DOS and Windows (3)
    Name                           : 1.ini

Attribute: 3
    Type                           : $DATA (0x00000080)
    Data size                      : 723 bytes
    Data flags                     : 0x0000

TODO: determine if $LogFile could be used to recover from this corruption scenario

LZNT1 compressed block with data size of 0

Not sure if this is a corruption scenario or a data format edge case.

A compression unit (index 30) consisting of the following data runs:

reading data run: 60.
data run:
00000000: 11 01 01                                           ...

value sizes                               : 1, 1
number of cluster blocks                  : 1 (size: 4096)
cluster block number                      : 687143 (1) (offset: 0xa7c27000)

reading data run: 61.
data run:
00000000: 01 0f                                              ..

value sizes                               : 1, 0
number of cluster blocks                  : 15 (size: 61440)
cluster block number                      : 0 (0) (offset: 0x00000000)
        Is sparse

Contains the following data:

a7c27000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
...
a7c27ff0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

This relates to an empty LZNT1 compressed block.

compressed data offset                    : 0 (0x00000000)
compression chunk header                  : 0x0000
compressed chunk size                     : 1
signature value                           : 0
is compressed flag                        : 0

It was observed in 2 differnt NTFS implementations that the entire block is filled with 0-byte values.

TODO: verify behavior of Windows NTFS implementation.

Truncated LZNT1 compressed block

Not sure if this is a corruption scenario or a data format edge case.

A compression unit (index 0) consisting of the following data runs:

reading data run: 0.
data run:
00000000: 31 08 48 d8 01                                     1.H..

value sizes                               : 1, 3
number of cluster blocks                  : 8 (size: 32768)
cluster block number                      : 120904 (120904) (offset: 0x1d848000)

reading data run: 1.
data run:
00000000: 01 08                                              ..

value sizes                               : 1, 0
number of cluster blocks                  : 8 (size: 32768)
cluster block number                      : 0 (0) (offset: 0x00000000)
        Is sparse

Contains the following data:

1d848000  bd b7 50 44 46 50 00 01  00 01 00 40 e0 00 07 0b  |..PDFP.....@....|
...
1d84c000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
1d84fff0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

This relates to a LZNT1 compressed block that appears to be truncated at offset 16384 (0x00004000).

compressed data offset                    : 16384 (0x00004000)
compression flag byte                     : 0x00

Different behavior was observed in 2 differnt NTFS implementations:

• one implementation fills the compressed block with the uncompressed data it could read and the rest with with 0-byte values
• another implementation seems to provide the data that was already in its buffer

TODO: verify behavior of Windows NTFS implementation.

References

• How NTFS Works, by Microsoft
• Master File Table, by Microsoft
• NTFS Attribute Types, by Microsoft
• File Attribute Constants, by Microsoft
• Reparse Tags, by Microsoft
• NTFS documentation, by Richard Russon
• ATTRIBUTE_LIST_ENTRY structure, by Microsoft
• ATTRIBUTE_RECORD_HEADER structure, by Microsoft
• FILE_RECORD_SEGMENT_HEADER structure, by Microsoft
• MULTI_SECTOR_HEADER structure, by Microsoft
• REPARSE_DATA_BUFFER structure (ntifs.h), by Microsoft
• REPARSE_DATA_BUFFER_EX structure (ntifs.h), by Microsoft
• USN_RECORD_V2, by Microsoft
• Zone.Identifier Stream Name, by Microsoft
• the Internet Explorer URL security zone, by Microsoft
• ntfs_layout.h, by Anton Altaparmakov