Specific Hard Drive Support

Topics: Users Discussion
Dec 15, 2015 at 12:28 PM
I need to buy some larger hard drives, as I am now desperate for storage space.

Obviously I would never use a hard drive without first encrypting it with VeraCrypt :) Once you have encrypted your data it is very difficult to go back to storing unencrypted isn't it ? :D

I have to admit to not fully understanding the problems VeraCrypt is having with large hard drives. All I know is there is some issue with 4K byte sector sizes.

I was hoping to buy these drives... Hitachi Deskstar 4TB NAS 7K4000 7200RPM SATA 6Gb/s 64MB Cache HDD - OEM (0S03356)

I am unable to find out if they support 512e, as there seems to be many model / serial numbers for what seems to be the same hard drive.

The data I wish to store is very valuable to me personally, I cannot risk losing it, which is why I am buying these additional drives.

From what I have read here, there is a chance VeraCrypt may work on large drives, or there is at least a work-around. As this data is so valuable to me I cannot take a chance that VeraCrypt could corrupt it in anyway.

Could someone please confirm / deny if I might have trouble using VeraCrypt with these drives ?

My set up and intention.

Win7 64 SP1
Latest VeraCrypt
Hitachi Deskstar 4TB NAS 7K4000 7200RPM SATA 6Gb/s 64MB Cache HDD - OEM (0S03356) as slave drives.
I intend to WDE (Whole Drive Encrypt) the Hitachi drives as slave drives (not boot drive) and format as NTFS.

Thank you for any help.
Dec 17, 2015 at 7:01 AM
What issues about 4k are you talking about? (source? references please)

What I know is, that some external drives emulate 4k sectors causing problems when you put
the hdd from the external case to an normal internal SATA port, as the drive itself uses 512 byte sectors.

This is not an VC issue - its just because of the controller of the hdd case.

If your data is valuable, you should have a backup anyway!
Dec 18, 2015 at 3:56 PM
A selection of tickets, some of which link to other forum posts, the issue does not appear isolated to external drives only.

https://veracrypt.codeplex.com/workitem/294

https://veracrypt.codeplex.com/workitem/277

https://veracrypt.codeplex.com/workitem/85


I am a humble user, not a programmer and I am left feeling unsure if VeraCrypt can support these large 4TB drives or if I may experience data corruption over time.

If VeraCrypt simply refused to encrypt the drive then I guess I could just gamble my money on these drives and test it myself. However what I am really afraid of is data corruption over a long period time which I am unaware of until it is too late. VeraCrypt may initially encrypt the drives and I may copy my data to it, only later to discover corruption has occurred.

These drives will be bought as the backup. Simply saying I should have backups isn't really fair, as these drives are being purchased to be the back up solution.

To add further detail to the above, these drives will be connected via internal SATA as slave drives. They are not and never will be fitted into external enclosures.

Does anyone know about this issue and if there is a simple test a humble user can perform to check for data corruption ? I will not buy large drives if I am unable to encrypt them with VeraCrypt, however for practical purposes I would like to be able to use one drive for all data and simply mirror it when backing up.

I looked for a help guide for large slave drives and VeraCrypt but I was unable to find one. One which talked the user through how VeraCrypt might need things setting up etc. I understand VeraCrypt needs drives with legacy support for 512 but there is no explanation how to force the drive to use it.

Any help would be gratefully received as time is becoming an issue for me, due to a desperate lack of drive space :) LOL
Dec 20, 2015 at 8:06 PM
Interesting references. I actually use VC with a physical 4k drive, but with 512 emulation and I have had no problem so far.
The problem is, that HDD can always have errors in old data, encrypted or not (keyword: silent data corruption).
You can detect those by using checksums and correct them by using redundancy.

sadly, common file systems (fat, ntfs, ext) do not support redundancy and checksums out-of-the-box.
Using Linux you can use btrfs which provides checksums, otherwise you can use file-based checksums.
The problem is: you cannot detect errors that happend before hashing. So if you checksum your backup
you don't know if tehre was an error transfering data TO the backup.

I'm not aware of a good file-based tool for redundancy. par2/par3 is a well-known example, but it is limited to at most ~32k files for performance reasons - which would mean one has to segment the data in groups for adding redundancy
Dec 21, 2015 at 3:43 AM
Edited Dec 21, 2015 at 3:43 AM
I suggest creating a local fileserver using an old computer and NAS4free (http://www.nas4free.org/), which will give you huge filesystem capacity with mondo redundancy in whatever RAID flavor you prefer (http://www.icc-usa.com/raid-calculator.html), and then using VeraCrypt to create encrypted file containers which you can store on and locally access from the NAS4free fileserver via your local (Gigabit Ethernet) network.
Dec 22, 2015 at 4:23 PM
Edited Dec 22, 2015 at 4:24 PM
Thanks guys for your reply. I appreciate your help.

However I was really only wanting to know if VeraCrypt is getting closer to being able to support large hard drives. Particularly those with no legacy 512 support.

I am also concerned if there is any way VeraCrypt may accidentally corrupt files on large drives which do support legacy 512e in a way the user may be unaware of. The following quote from Mounir makes me nervous.
The modification of VeraCrypt on Windows to support sector size other than 512 will take some time. Till then, big disks with no 512-bytes emulation (e.i. 4K native) will have stability issues on Windows.
Until there is an official 4k support in Veracrypt, should humble home users such as myself stick to smaller hard drives if they wish to use VeraCrypt ? 4TB drives are becoming affordable to home users now so I can see a demand for 4k support growing quickly.

I am always worried about "emulation". If these drives listed above, do offer emulation they may still not work in the way VeraCrypt is expecting, resulting in data corruption.

I am only nagging about this because I have reached the limit on my current drives, I dare not buy any replacements until I am certain I am able to use VeraCrypt on them. I could buy 4 of the above 4TB drives (preferred option) or do I have to buy 8 smaller 2TB drives to ensure compatibility with VeraCrypt ?
Dec 23, 2015 at 7:38 PM
I was surprised when I read these comments. I have numerous of both internal and external 4TB drives from WD and Seagate that have been encrypted with VeraCrypt for over a year now. I use them on a daily basis and have had no problems. More recently I have encrypted 5TB Seagate drives, 6TB WD and Samsung drives, and one month ago I encrypted a pair of 8TB Seagate drives. Since I have had no problems with any of these drives after so much use, I don't know how to evaluate any ongoing risk. I have all my data backed up onto similarly sized and encrypted drives so... should I worry?

To give a little further info, most of the encryption was done while using Win7 but I now use Win10. After receiving the drives I delete the existing format it comes with and convert to GPT. I then usually create a small leading partition of a few GB and then do a 'quick format' of the remaining space. I then use VeraCrypt to encrypt the large partition.
Coordinator
Dec 23, 2015 at 8:20 PM
Hi all,

Just to put the issue in context: all the reports received until now concern disks with no partitions (RAW mode encryption) and only if the 512e emulation mode is not active. The other issues posted above by DBKray have nothing to do with 4K support (one is a failure to format internally using NTFS because of instable behavior of a Windows API that we use but formatting can be done manually, and the other concern a RAID controller with disks that use 512e emulation mode).

Many large disks still come with 512e mode (easy to check using fsutil command) and avoiding encryption of RAW disk should be enough.

As I explain previously, the issue is that the driver code always uses 512-byte alignment for data read/write and this can cause issue on 4K disk controllers if Windows sends buffers of data that are not 4K aligned.
Luckily, partitions are usually 4K aligned and if the file system used in the volume is NTFS with a cluster size that is multiple of 4096 (which is the default) then all read/write operations should be 4K aligned.

Of course, this issue must be addressed to avoid issues in special cases. I have already done some modifications but since this part of code is very sensitive, I will not release anything before doing extensive testing and debugging.
Dec 28, 2015 at 2:23 PM
Thanks Mounir

I have become so desperate for storage space I have already ordered the drives. I thought that if there are any problems I could use smaller volumes on these large drives.

However, as Mounir has explained it is only related to RAW encryption then I guess I should be ok :)

This captcha thing is a nightmare! Unreadable!
Dec 29, 2015 at 8:24 PM
Just create aligned partition(s) and you should be fine, I think that even GPT partitions worked on an additional drive, only system encryption is not possible with GPT. Captcha sucks, but when you reload, it displays usually better image.
Apr 30, 2016 at 10:08 PM
Sorry to return to this thread again and so late, but believe it or not I have only just received these drives, it's a long storey !

Monuir
and avoiding encryption of RAW disk should be enough.
I have realised that "RAW" mode is what I was intending to use, as I wanted to encrypt the entire slave drive and not simply make a container file.

Monuir
Of course, this issue must be addressed to avoid issues in special cases. I have already done some modifications but since this part of code is very sensitive, I will not release anything before doing extensive testing and debugging.
Has there been any progress on this recently ? I would very much like to use RAW mode on these 4TB drives as soon as you say it is safe to do so.

Also I am unsure how to format the opened drive, as far as I understand NTFS cannot, or perhaps should not, be used on drives larger than 2TB. So I suppose I need to RAW encrypt the entire drive, open it and then format the opened drive as GPT. I guess I need to do this manually through windows disk management after encrypting with VeraCrypt ?

Thanks for your work on VeraCrypt Monuir.
Coordinator
Apr 30, 2016 at 10:17 PM
Thanks for the update.

I think there is a misunderstanding about the word "RAW". By RAW, I mean a non-partitioned disk where a drive is encrypted in whole without first create a partition on it. The opposite to "RAW" is not using file containers but rather creating a unique partition on the disk that will take all free space.

In this context, do you really need to encrypt the disk in "RAW" mode without any partition or is it acceptable for your to create a unique partition on it (GPT) and then encrypt this partition using VeraCrypt?

As for using NTFS, you can choose it as filesystem even for such big disk. During the disk encryption process in VeraCrypt, just choose NTFS and VeraCrypt will do it for you.

Anyway, as I said earlier, partitioning your disk and encrypting this partition alongside using NTFS for the filesystem should be enough to avoid any issues.

Concerning the modification in VeraCrypt to adapt to native 4096 bytes alignement, no advancement has been made on this since the priority now is UEFI support which is advancing quite well and an experimental version should be available very soon.
Apr 30, 2016 at 11:17 PM
Wow Monuir, I was not expecting such a rapid response this time of night :) I have been web surfing to waste time not thinking you would be online. Thanks for your reply.

I am almost sure I understand "RAW". I use it to make the disk look completely random, like it has been wiped with a drive wipe tool using random characters.

I am using RAW in the hope it may provide a small chance of plausible deniability. I appreciate skilled forensic people may question it, but I am not trying to protect it from those types.

I do the following normally, on 2TB or less drives.

Working through the VeraCrypt menu...
Press Create Volume.
Encrypt a non system partition drive.
Standard Veracrypt Volume.
Create encrypted volume and format it.

Then enter my password and encrypt.

When finished the unmounted drive looks like random data from the beginning to the end, no obvious partition.

I assume this is "RAW" mode ? I don't feel comfortable having partitions with my encrypted disks. I realise there are downsides to this approach and it is also not 100% convincing to experts, but I do feel better doing it this way.

I fully appreciate all the work you are doing to VeraCrypt and you have prioritised jobs to do in your own way, however if you would please remember the 4k issue and perhaps incorporate it onto your wish list I would be very grateful.

A suggestion, perhaps it might be a good idea to remove the option to RAW encrypt drives larger than 2TB until you have had time to test and modify VeraCrypt with 4k ?

Thank you very much for your help :)
May 1, 2016 at 1:49 PM
OK, I am so desperate for drive space I need to use these disks and I guess I will have to go without the plausible deniability for a while.

Can someone walk me through this please ?

I have connected the drive

opened computer management

Made the disk online

Converted the drive to GPT

The disk correctly shows 3725.90GB

I right click on the unallocated space, select "New simple volume".

Choose the maximum size for the volume in the wizard.

Assign drive letter.

Here is where I get stuck, I am asked if I want to format the partition using NTFS. Do I need to format it if I am going to use VeraCrypt to encrypt the entire partition next ?

Thanks for any help.
Coordinator
May 1, 2016 at 2:08 PM
No need for formatting since it will be oerwritten anyway by VeraCrypt encryption.
Just create the partition and go to VeraCrypt to encrypt it and you can select NTFS formatting in VeraCrypt encryption wizard.
May 1, 2016 at 2:26 PM
Thank you very much Monuir for the advice, you are doing a wonderful job with VeraCrypt. I just wish there were more people as clever at programming as you so you had some help.

I am very grateful for your contributions to personal privacy and for protecting my data from criminals. I just wish a large company or organisation would sponsor you.
Coordinator
May 1, 2016 at 3:04 PM
Thank you for your kind words. I'm just doing my best with my limited resources to help people protect their information.

Indeed more help is needed on the development side but also on the support side: the user base is growing and more people with technical expertise on VeraCrypt and its usage are needed to investigate reported issues and propose help.

Of course, the development pace is slow and there are so many other things to do. Without financial support, things will not get better. I double that any sponsoring will come an time soon. As I said before, the only solution seems to be building an enterprise specific extension or service that could secure some funding for the future development.
May 7, 2016 at 1:41 PM
Sorry for my late reply, I immediately started to encrypt these 4TB drives and then load them with data, which takes a VERY long time :)

Just to report everything seems ok with 4TB drives when not using RAW whole disk mode.

As soon as you say VeraCrypt supports 4k and it is safe for VeraCrypt users to use RAW mode on large drives, I will re-encrypt them. Although not 100% protection from a professional investigator, I really like the little extra plausible deniability RAW mode offers.

I wonder if we could encourage the EFF to provide some financial support for you ? VeraCrypt has to be one of the most significant privacy tools for computer users and it should be adopted. Personally I rate VeraCrypt more significant than GnuPG and I love GnuPG !

Another idea is to ask for sponsorship by private companies who could perhaps have their names listed in the "About VeraCrypt" box and even advertisements during installation.

I am ashamed at how little I and others have helped you financially.

Please keep up the wonderful work Monuir, there are many grateful people wishing you well, even though they don't post on this forum.
May 23, 2016 at 1:24 PM
Hi Monuir

I thought it a good idea to test one of these drives in RAW mode and then run fsutil fsinfo ntfsinfo on the mounted drive letter.

I am very keen to use RAW mode on these drives as plausible deniability is important to me.

Monuir could you please take a look at the output for the mounted drive letter and let me know if it is safe to use these particular drives with RAW mode ?

Number Sectors : 0x00000001d1c0bcaf
Total Clusters : 0x000000003a381795
Free Clusters : 0x000000003a374d6b
Total Reserved : 0x0000000000000000
Bytes Per Sector : 512
Bytes Per Cluster : 4096
Bytes Per FileRecord Segment : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length : 0x0000000000040000
Mft Start Lcn : 0x00000000000c0000
Mft2 Start Lcn : 0x0000000000000002
Mft Zone Start : 0x00000000000c0040
Mft Zone End : 0x00000000000cc840


I am guessing that "Bytes Per Sector:512" is the important part ??? If so, it seems to me that I can use these drives in RAW mode but I would very much appreciate your advice.

Thank you.