New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
I have read it (as it was happening, all of it with the long non stop posts). My point is still the same. I am not disagreeing with you in theory and practice, I am disagreeing with the intentions behind the thread and its initial wording. Other than that, we have no real differences.
Wow, the idiot post got to you eh? I guess even idiots know, somewhere deep inside, that they are idiots. At least that answers a few questions I had. I'll take it easy on you idiot from now on.
Nah, definitely I will not prolong discussion with you. Clueless old man. Period.
Idiot
Can you please explain this to me? I want to understand what's your point here?
I don't think I can make it clearer.
It started in another thread, and then a new one was created, which, IMO appears to be a thread that was designed to become, well, this.
So, I took some time and effort to run my tests on a public cloud provider so that anyone could test it. However, @jsg interpreted the very fact that it was done on a public cloud provider as an indication that the public cloud provider might have different IOPS characteristics, although this absolutely wasn't the case through the fio tests immediately proved him otherwise.
He (like many others) isn't willing to spend money on cloud providers, so I'm working on something that everyone can test using virtualization software, at their laptops or desktops, no "evil" Amazon required. Stay tuned.
~ETA?
FWIW: My position on the Contabo NVMe has changed, see here -> https://www.lowendtalk.com/discussion/comment/3273873/#Comment_3273873
The reason is that some facts have changed and there is new data to back it up. To avoid misunderstandings: the reason is absolutely not some mischievous users here and their "I don't care if others are bleeding. I'm here for what I take to be fun" gang.
I follow the data. If they change, my view changes.
Monday™, possibly sooner.
There is a valid reason to stick with one tool for a certain baseline. Different tools do different workloads even when the same testing technique is applied. The problem will still remain that one tool performs better on hosting X then some other tool on Y.
This is not unique to this thread and happens pretty much with everything and it's healthy to discuss difference between testing tools. I wouldn't have my own testing tool though, use something proven community driven
So, the problems with vpsbench that I pointed out can be reproduced outside of Amazon as well. My new VM based test is described below.
What we're testing
The point of contention, is whether vpsbench can be trusted to accurately report and be a good indicator of disk performance. The best way to test this is to take a disk with known performance characteristics.
I did this using Amazon gp3 disks limited at a lower bound of either 3000 IOPS or 125 MB/s, which did demonstrate vpsbench reporting numbers that were way off. @jsg disagrees with this methodology as he believes that a virtualized storage framework (which Amazon is surely using to offer gp3 disks) could hard limit I/O, but this was proven to him with fio benchmarks.
As this was still deemed unsatisfactory (unfortunately, with unsubstantiated name calling and theoretical hypothesizing), so I'll use a much less controversial way to limit I/O: native mechanisms of the Linux kernel. Specifically, I'll use Docker to place these limits. The only problem is that this experiment would be much less tightly controlled as not everyone has the exact same hardware as me, but the general issue should be reproducible on any modern system.
Specifically, this test will emulate a slow disk by instructing the OS to slow down or deprioritize I/O request from the Docker container, whereas in the initial test, the gp3 disk presented a disk that appears to the OS as slow. This difference should be insignificant though, as all userspace applications (applications that require an OS to run) have to pass I/O requests through the kernel, and a slow disk would appear the same as the OS slowly processing I/O requests.
Prerequisites
As mentioned, this experiment is far looser than the first one because it has to be reproducible on any modern Linux system. However, in the interest of transparency, this is what I used to test:
The tests
To get an idea of what kind of I/O performance the VM itself has (I'll keep referring to our system as a VM from now on as that is where I tested), I'll begin off with two fio tests, a 4k and a 1m random read-write(50%-50%) test. Make sure to run these multiple times to get an accurate estimate of performance.
In my case, I ended up with these, which should be fairly representative given my other runs:
From the 4k test, we see that our disk can handle a minimum of 10560 read-only/write-only IOPS as per the the 4k test, and a minimum throughput of 620MB/s for the 1m test. This gives us a maximum threshold of how much I/O our VM can safely handle.
Now I'll create a Docker container that's limited to a low value of IOPS that is below what you've inferred from the above tests. In the command below, I limit it to 1500 IOPS and a throughput of 65 MB in each direction (read/write):
For YABS to work (you could also use fio directly, but YABS just a thin wrapper around fio anyway), I install fio. This step is technically not required, but then I'd have to fudge around with curl so that YABS can download fio. As a nice side effect, since both the VM and the container are on Ubuntu 21.04, it keeps the fio version the same on both:
Now, if I run vpsbench, this is what I get:
Unlike the Amazon EC2 test, where the Wr Seq/Wr Rnd values were somewhat off and Rd Seq/Rd Rnd was way off , in this environment, vpsbench manages to detect the I/O limits pretty closely for the first three tests (69 MB/s as opposed to 65 MB/s). But then we have the fourth value which, is way off similar to my original post. It also defies any reasonable explanation -- how does a random read test manage to outperform a sequential read test?!
Here are the YABS results for comparison:
These tests are quite close to what's expected. The maximum IOPS detected is slightly lower [2.9k instead of 3k] and the throughput is also slightly off, but with a lower margin of error than vpsbench does.
I've run these multiple times to ensure that these results are not one-offs, and you're encouraged to walk through them yourself.
Conclusion
You cannot generally trust vpsbench to accurately represent disk speeds and especially read speeds. If you use the Linux version, you could trust the write speeds although based on the environment, they can have a fair margin of error.
@stevewatson301 well it's just fio's version of truth and vpsbench's version of truth.
Choose which truth you trust and keep it simple.
The truth is out there.
@stevewatson301
Your first problem is that you start on a bunch of assumptions, e.g. that Docker, virtualbox, or whatever actually respect -and- can enforce the limits you set. This however is only true to some degree, because NO OS or hypervisor really tightly controls limits, emphasis on 'tightly'. Simple reason: it doesn't come for free hence the question arises "where on the axis do we want to be?" or in other words, how much time and how tight a control loop are we willing to spend in order to achieve some level of tight control? As always with that kind of question, and they arise very often in computing, there's some trade-off and one must decide where on the axis between the two extremes ("No control at all, no cost at all" vs. "Perfect control, very high cost") one wants to be.
Gladly, benchmarks don't care much, they just work with and measure what is available, be that GB per second or be it KB/s - just like the applications users run.
Two points that you seem to consistently fail to understand:
(a) benchmarks are different and you just can't take benchmark A as "the reference" to judge benchmark B.
vpsbench works differently from fio and particularly yabs. For example, unlike yabs which uses AIO and even multiple tasks, vpsbench intentionally uses single-threaded testing and standard IO. Why? To name just one reason, the vast majority of programs in the vast majority of cases do not use AIO and it's of questionable value, especially on LET (hint: low end [boxes and] talk) to use multithreading when many systems under tests only have 1 vCore and often not a lot of memory. Also while multi-threading overhead has significantly decreased on modern OSs it still doesn't come for free. That's why vpsbench only uses it in network testing, where a really significant blocking time is to be expected.
(b) A weird result, if obtained by well time proven code, doesn't mean that the code or algorithm is bad. It simply means that there is something out of the ordinary behind the scenes, often an actually interesting mechanism.
And indeed I like the fact that vpsbench does detect those things! Where you see a problem or defect I see an interesting and in fact often valuable hint, typically pointing something out to me (e.g. about the caching of the node).
I don't want to talk about fio, especially not bad; I'm confident that it does provide what the author wanted to get with his tool. But I - intentionally - use another approach, namely a very simple, "boring" standard mechanism ("read or write X bytes Y times") and focus on precise timing results. Plus, as I already said many times, I intentionally try to be a good neighbour by pausing between "blocks"/slices; it's just a few milliseconds but that makes all the difference between behaving like a pig on a VPS or like a gentleman. But of course some milliseconds is a lot of time on a fast system, enough time, for example, to update many caches.
As for what you complain about, the "weird" random read result, that's relatively simple to explain: given the memory is available a modern OS know that it needs to cache large chunks or if possible even the whole file. With a test file size of 2 GB that's well feasible. The result is the "weird" high value you see because unlike with sequential reading, the read ends up in a large cache most of the time rather than on the drive. fio (as used by yabs) doesn't see that because it works quite differently. That doesn't mean that yabs/fio is a bad benchmark - but nor does it mean that mine is bad. They are different and had different goals.
Btw. keep in mind that AIO doesn't really make the IO faster. AIO is meant to have its callers (usually programs or libraries) waste less time on blocking.
You can see in his previous tests that the YABS result was in line with the limits he set, which proves what you said wrong.
@drunkendog if the OS, hypervisor can really tightly controls limits, emphasis on 'tightly', it's just another version of truth.
Well I see that version of truth a lot, on nearly a hundred of servers I'm maintaining. But maybe it's just me, maybe a different version of truth exists around.
No, it doesn't. (a) if it proves anything then it proves that the limiting mechanism worked better with the way yabs/fio does its testing than with the way vpsbench does its testing. And btw, even that is questionable because those limits are on IO but not about OS caching. And (b) we are talking about a rather small difference, about 66 MB/s vs. about 69 MB/s which is easily explained by my tiny pauses between reads/write allowing for some cache filling or in fact even by the different ways the 2 benchmarks do their testing (e.g. multiple reads/writes vs single reads/writes).
Btw, (most) users are not interested anyway in how well IO can be limited. They are interested in how much IO they can get/expect.
And your comment doesn't show my comment you quoted wrong in any way.
So my recommendation would be to not perform extremely small slices of tests and linearly extrapolate them to very large values, as this is proving to be quite the incorrect way to infer performance of a system.
If you want more evidence for why this is the case, get one of those Oracle free tier instances with their 50 MB/s caps -- quite slow for any real task, but this is what the value of random read is reported as on those instances:
And it fails to do that, because most providers, including those on LET, have I/O caps, otherwise people would trash the disk to shit with torrents and Chia. Also, @Falzo even provided a test from his dedi that shows near RAM speeds, which should tell you that something with your benchmark is not quite right?
When placing I/O limits at the OS level, all I/O syscalls are slowed down. And with fio,
direct=1(or O_DIRECT in Linux syscall terms) should bypass the OS caches.For this test, the issue is with the Rd. Rnd value and even I said as much.
Of course, on that first with AWS test both read values are inaccurate, so this is not a recommendation that one can trust either of the Rd Seq and Rd Rnd values.
Not sure why you bring up this difference, I just happen to use the aio engine but of course with the kinds of I/O tests being performed, I believe it can be swapped out for any other implementation as none of the disks that I'm using (or the I/O limits that I've placed) have sufficient performance for these differences to be significant enough.
One can do tests with small slice sizes, medium slice sizes, large slice sizes ... just like with fio (and many other benchmark tools)
I have been asked how vpsbench works and I have laid out how it works. All tests, read or write, sequential or random, buffered or sync use exactly the same routine, except for the random tests not working sequentially but with random locations.
Yet you either ignore that or don't grasp it but stubbornly make statements that are definitely false. And of course you pick the results that fit your agenda, btw not even understanding that those "weird" values actually contain interesting information.
And again you seem to not really understand what you are talking about. For a start those calls (mostly) bypass the guest VMs caches but not the nodes caches. Plus there is a difference between writing and reading. Plus fio's (as used by yabs) way of working is radically different from vpsbench's; of bloody course you get different results and of bloody course throwing 64 or whatever calls basically simultaneously at the OS vs. doing one call at a time leads to very significant changes in cache performance.
So, if a user happens to be interested only in some specific use case, say for a database, using fio with the right parameters is better than vpsbench, but I don't care because I'm after the performance a user can expect generally.
There is only one thing so far I see worth of some consideration: vpsbbench - until now - only tested with one (1) block/slice size. I understand that it would be desirable to test with a small (4 KB), a medium (64 KB), and a large-ish (1 MB) block size. And that "lesson" I not only learned from listening to feedback (not yours though) but I already implemented it in my newest version (another good example of the utterly idiotic and nasty allegation that I never listen to user feedback, yada yada, being ridiculously wrong).
I'm getting tired by you again and again turning your lack of understanding into alleged weaknesses of my vpsbench. Plus you totally ignore anything not fitting your private war.
Please link to where you have actually ran the program and concur with the results of vpsbench? I have tested it and explained in detail the problems with the results for 1-2 years. Wtf makes you think LET gives a flying fuck about your useless opinions?
15 visits in 6 years? Reading all those threads? And no alt accounts? I call shenanigans.
This isn't a case of two watches, one slightly fast and one slightly slow. This is one being slightly off and one being the wrong hour and minute.
I think it's more like 2 watches with different methods of timekeeping - contradicting each other, in this analogy..
OMG, this screams "you're doing it wrong". It's especially egregious considering all the boasting about all the caching testing you've done and supposedly do things to avoid caching affecting results (if I wasn't on my phone, I'd dig up your previous posts on this topic).
You're saying you're perfectly ok with a disk benchmark where NO data is read from storage media (since its completely in cache, which is defective test setup in the first place). That's facepalm fail.
(Also, a random read test uses a single 2GB file?!?!)
Who made you the LET King you dumbass hillbilly.
That sounds like an MJJ answer
(Our clock and calendar, all else can GFT).
Please use proper punctuation when making insults, especially a hillbilly one. eyeroll
Hillbillies are an American thing. If Letterkenny is to be believed the Canadian term is 'hicks'. Rednecks however are geographically unrestricted. Go with that next time.
Edit: typo
Ok, you dumbass Redneck
Actually I think redneck is more offensive as it has racial undertones to it.