Toshiba disk low performance

Discussion:

(too old to reply)

Christos Chatzaras

2018-06-07 15:02:03 UTC

I order a new dedicated server from a provider. The server has 2 Toshiba disks and the disk peformance is very slow.

I test it using mysqlcheck -Ao which optimizes the databases and also with 10GB mysql import at 2 servers - one with the Toshiba disks and the other with Seagate disks.

During the tests gstat for the server with Toshiba disks show 3-4 times lower disk operations / second compared to the other server.

I had the same problem (same Toshiba models) in a production server (I notice it after I put it in production) and ask the datacenter to replace one disk with different model, I rebuild RAID-1, then replace the second disk and I rebuild the RAID-1. And the issue resolved.

It shows these disks as UDMA5 but other disks with good speed show UDMA6. Also Linux shows these disks as UDMA5.

Here is the dmesg output:

----
ada0 at ahcich1 bus 0 scbus0 target 0 lun 0
ada0: <TOSHIBA MG04ACA400E FP3B> ATA8-ACS SATA 3.x device
ada0: Serial Number Z5B7K1Q4FJKA
ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 3815447MB (7814037168 512 byte sectors)
ada1 at ahcich2 bus 0 scbus1 target 0 lun 0
ada1: <TOSHIBA MG04ACA400E FP3B> ATA8-ACS SATA 3.x device
ada1: Serial Number Z5B7K1Q5FJKA
ada1: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 3815447MB (7814037168 512 byte sectors)
----

What could be the problem with the Toshiba disks?

Christos Chatzaras

2018-06-07 15:18:17 UTC

Permalink

During a 10GB mysql import the server with Toshiba disks does 250-400 ops/s and another server with exactly the same setup (only disks different) does 1000-2300 ops/s.

When servers are idle "diskinfo -ctv /dev/ada0" show similar speeds to both disks. During the mysql import the server with Toshiba disks show 15 times worst results.

Post by Christos Chatzaras
I order a new dedicated server from a provider. The server has 2 Toshiba disks and the disk peformance is very slow.
I test it using mysqlcheck -Ao which optimizes the databases and also with 10GB mysql import at 2 servers - one with the Toshiba disks and the other with Seagate disks.
During the tests gstat for the server with Toshiba disks show 3-4 times lower disk operations / second compared to the other server.
I had the same problem (same Toshiba models) in a production server (I notice it after I put it in production) and ask the datacenter to replace one disk with different model, I rebuild RAID-1, then replace the second disk and I rebuild the RAID-1. And the issue resolved.
It shows these disks as UDMA5 but other disks with good speed show UDMA6. Also Linux shows these disks as UDMA5.
----
ada0 at ahcich1 bus 0 scbus0 target 0 lun 0
ada0: <TOSHIBA MG04ACA400E FP3B> ATA8-ACS SATA 3.x device
ada0: Serial Number Z5B7K1Q4FJKA
ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 3815447MB (7814037168 512 byte sectors)
ada1 at ahcich2 bus 0 scbus1 target 0 lun 0
ada1: <TOSHIBA MG04ACA400E FP3B> ATA8-ACS SATA 3.x device
ada1: Serial Number Z5B7K1Q5FJKA
ada1: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 3815447MB (7814037168 512 byte sectors)
----
What could be the problem with the Toshiba disks?
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-questions

Christos Chatzaras

2018-06-07 15:24:36 UTC

Permalink

I also see this differences between other server disks and server with toshiba disks. Toshiba disks show PIO 8192bytes and the other "good" disks show PIO 512bytes.

Toshiba disks:

ada0 at ahcich1 bus 0 scbus0 target 0 lun 0
ada0: <TOSHIBA MG04ACA400EY FQ1B> ATA8-ACS SATA 3.x device
ada0: Serial Number 382RK075F7GB
ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 3815447MB (7814037168 512 byte sectors)
ada1 at ahcich2 bus 0 scbus1 target 0 lun 0
ada1: <TOSHIBA MG04ACA400EY FQ1B> ATA8-ACS SATA 3.x device
ada1: Serial Number 383IK043F7GB
ada1: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 3815447MB (7814037168 512 byte sectors)

"Good" disks:

ada0 at ahcich1 bus 0 scbus0 target 0 lun 0
ada0: <ST4000NM0245-1Z2107 SS03> ACS-3 ATA SATA 3.x device
ada0: Serial Number ZC112ALJ
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 3815447MB (7814037168 512 byte sectors)
ada1 at ahcich2 bus 0 scbus1 target 0 lun 0
ada1: <ST4000NM0245-1Z2107 SS03> ACS-3 ATA SATA 3.x device
ada1: Serial Number ZC111JK1
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada1: Command Queueing enabled
ada1: 3815447MB (7814037168 512 byte sectors)

Steve O'Hara-Smith

2018-06-07 16:19:33 UTC

Permalink

On Thu, 7 Jun 2018 18:02:03 +0300

Post by Christos Chatzaras
I order a new dedicated server from a provider. The server has 2 Toshiba
disks and the disk peformance is very slow.

I have had similar experiences with Toshiba drives, it improves
somewhat when they have finished remapping all the bad sectors IME.

--
Steve O'Hara-Smith <***@sohara.org>

Christos Chatzaras

2018-06-07 16:54:56 UTC

Permalink

I test it with 3 different servers with Toshiba HDDs. The disks are new, only few hours in usage. I don't think they have any bad sectors.

Post by Steve O'Hara-Smith
On Thu, 7 Jun 2018 18:02:03 +0300

Post by Christos Chatzaras
I order a new dedicated server from a provider. The server has 2 Toshiba
disks and the disk peformance is very slow.

I have had similar experiences with Toshiba drives, it improves
somewhat when they have finished remapping all the bad sectors IME.
--
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-questions

Valeri Galtsev

2018-06-07 17:05:29 UTC

Permalink

Post by Christos Chatzaras
I test it with 3 different servers with Toshiba HDDs. The disks are new, only few hours in usage. I don't think they have any bad sectors.

This is exactly why we have virtually no Toshiba hard drives around.
Also, neither of vendors who build machines which we buy uses Toshiba
drives. And once we notice hardware from some vendor does not behave
decently over time (about 6 years is a must), we abandon such vendor for
good.

"Pricegrabber" is your enemy when hardware is concerned. Once someone
stated it so well, that I just keep repeating it. If you can spend
10-15% more but get definitely good reliable hardware, do it. Otherwise
you will pay much more with your time, and will pay to replace failed
hardware soon as well.

But alas, the market is driven by mass customer who does not care about
the above, and keeps paying what looks like much less for something that
is not worth it.

Valeri

Post by Christos Chatzaras

Post by Steve O'Hara-Smith
On Thu, 7 Jun 2018 18:02:03 +0300

Post by Christos Chatzaras
I order a new dedicated server from a provider. The server has 2 Toshiba
disks and the disk peformance is very slow.

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-questions

--
++++++++++++++++++++++++++++++++++++++++
Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247
++++++++++++++++++++++++++++++++++++++++

Christos Chatzaras

2018-06-07 17:42:26 UTC

Permalink

boniee++ shows lower performance on these Toshiba disks but not much.

Also untar big archives show the same execution time on Toshiba and no Toshiba disks.

But the issue happens doing "mysqlcheck -Ao" and "mysql import".

Any idea how to reproduce the issue with a benchmark tool?

Arthur Chance

2018-06-07 20:12:39 UTC

Permalink

Post by Christos Chatzaras
I test it with 3 different servers with Toshiba HDDs. The disks are new, only few hours in usage. I don't think they have any bad sectors.

Backblaze are my go-to source for disk performance. Looking at the
latest quarterly report
(https://www.backblaze.com/blog/hard-drive-stats-for-q1-2018/) although
they are trying out Toshiba's new 8TB drives, it's noticeable that they
only have 191 Toshiba drives deployed out of an entire fleet of 98,046
drives. This rather suggests they are not keen on Toshiba disks. You'd
need to look back through their quarterly reports to find out why though.

--
An amusing coincidence: log2(58) = 5.858 (to 0.0003% accuracy).

Christos Chatzaras

2018-06-08 11:42:52 UTC

Permalink

I did some benchmarks and the problem exist in both Linux and FreeBSD.

I benchmark a server with Toshiba disks and a server with Seagate disks.

First I install debian without RAID-1. Then I install iozone.

Tests done with hdparm to test disk bandwidth and iozone to test random read/writes.

Here are the results:

https://hastebin.com/gayidosaju.pas

As you can see the bandwidth for the Toshiba disks is slower than Seagate but this doesn't really matter.

The problem is the second test that does random read/writes which show the Toshiba has 10 times slower random write speed than the Seagate.

That's why the issue happens mostly during a "mysql import" and not other server tasks.

Daniel Feenberg

2018-06-08 14:02:31 UTC

Permalink

Post by Christos Chatzaras
I did some benchmarks and the problem exist in both Linux and FreeBSD.
I benchmark a server with Toshiba disks and a server with Seagate disks.
First I install debian without RAID-1. Then I install iozone.
Tests done with hdparm to test disk bandwidth and iozone to test random read/writes.
https://hastebin.com/gayidosaju.pas
As you can see the bandwidth for the Toshiba disks is slower than Seagate but this doesn't really matter.
The problem is the second test that does random read/writes which show the Toshiba has 10 times slower random write speed than the Seagate.
That's why the issue happens mostly during a "mysql import" and not other server tasks.

Assuming that Smartdisk doesn't show a problem, don't you think that for
some good or bad reason, the writes to the Seagate are async and writes to
the Toshiba are sync? Perhaps FreeBSD has some rules about the default
that Toshiba triggers in a different way than Seagate?

Daniel Feenberg

Christos Chatzaras

2018-06-08 14:45:58 UTC

Permalink

I run these benchmark tests in Linux which show slow random writes. I don't think the issue is related to FreeBSD.

Assuming that Smartdisk doesn't show a problem, don't you think that for some good or bad reason, the writes to the Seagate are async and writes to the Toshiba are sync? Perhaps FreeBSD has some rules about the default that Toshiba triggers in a different way than Seagate?
Daniel Feenberg
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-questions