FFMpeg benchmark - transcoding into H.264 with different presets

Couple of years ago I developed simple software for recording from security cameras for one hotel. For various reasons I had to pull the video from the cameras in MJPEG format (Motion JPEG). MJPEG quality/bitrate ratio is quite awful, so for long-term storage, I transcode it to H.264. However, there is a lot of parameters which can affect encoding time, size of the output video and quality. So now I finally have tool to check which one would work best for me and I hope the results will be useful also for some of you.

I benchmarked FFMpeg command-line software in version N-81475-gdc7e5ad from 27 August 2016. I used two different input videos and encoded each in H.264 using constant quality setting and different presets.

Preset is prepared collection of specific settings which specify details of encoding. Using different presets, one can change the encoding time/file size/quality ratio (more about presets: https://trac.ffmpeg.org/wiki/Encode/H.264).

I used constant quality setting, so we can see the difference in run time / file size while maintaining +- same quality. As per recommendation of the manual (using "sane" values), I measured transcoding to bad quality (crf=18), default quality (crf=23) and good quality (crf=28).

Each test was run from command line as:
ffmpeg.exe -i {input_file} -c:v libx264 -profile:v high -level 4.0 -preset {preset} -crf {quality} result.mp4

System parameters

The benchmark was executed on following system:

Laptop Acer TimelineX 5820TG
CPU:	Intel Core i5 M460 2C/4T @ 2.53 GHz
RAM:	4GB DDR3 1067 MHz
SSD:	Kingston SSDNow V300 128GB
OS:	Windows 10 (10.0.14393.105)

Results

H.264 video (Simpsons movie trailer, FullHD, 120 MB, 2 min 17 seconds)
MJPEG video (Camera recording, 640x480, 92.6 MB)

Conclusion

If somebody tells you slower preset will give you smaller file size, DON'T BELIEVE THEM (until they prove it to you). Unfortunately, it is not true, in all tests fast preset produced bigger size than veryfast preset, while taking much longer time on CPU. I re-measured this with couple of other videos, on three different computers and also on Linux, results were the same. I tried filling the bug first on FFMpeg, then on x264, but so far I got only responses which states that it is not a bug.

In current situation, all I can recommend is to use only ultrafast, superfast or veryfast preset, depending on how much time you have and how much you value your storage space.
Increase in constant quality setting adds some time to transcoding and the result file is much bigger, so bear that in mind when deciding what quality to use.

You can download all results from https://drive.google.com/open?id=0B43-CuEfEyUVZXlMRzNQVDRmNkE.

Follow-up on bechmarking 7-zip

Couple of notes on things I found out of during benchmarking 7-Zip:

• Each combination of parameters was run 4 times and then averaged. I always checked the deviation, so I would catch the runs during which OS did something on background and slowed the benchmark. And boy, that happened a lot! I had to rerun almost half of ZIP format runs, just because one run was off by 10%. Thanks a lot, Windows 10. 
• What I am particularly proud of is run with combination of ZIP format, highest compression level and 8GB image file. That took almost and hour and the difference between highest and lowest time was less than 1 second. It means less than 0.03% difference, which I think is quite impressive for machine which dynamically changes CPU speed (or just a coincidence, IDK).
• My poor old laptop spent more than 10 hours running just the results I published. However, the whole benchmark took more than 3 days, as I ran couple of more runs (see bellow), had to re-run some runs and shut it down during nights so we both get some sleep.
• Originally I wanted to include also test case with compressing video file. I used 120,43 MB video encoded in H.264 only to find out that the best compression 7-Zip can do is to 120,4 MB. So I guess H.264 is damn good optimized, 7-Zip tried for more than 80 seconds and it figured out how to make it smaller by just 0,025%.
• As for other archive formats, I did run some tests using XZ format only to find out the results are exactly the same as 7Z format. It would be nice to look at the difference between these formats in the future.
• (Most important for the future) Making graphs in Excel is really not an option for next time, I have to use some library for generating them automatically. So stay tuned for next benchmark with better graphs.

Edit 16 Sep 2016: As I have some DB schema changed in my mind, I might be truncating the DB in some time ahead. In order to preserve exact results from this benchmark ("for future generations"), I exported the data and uploaded it here.

7-Zip benchmark - 7Z vs ZIP

It took a little bit longer than I expected, but the first benchmark using Benchroom is done! So here it is!

The goal of this benchmark is to find out how various archive formats with different compression algorithms behave when compressing different types of input. I made the tests using 7-Zip (version 16.02 64bit), I tested ZIP and 7Z (LZMA compression) with different levels. Each test combination was ran 4 times and the results were averaged.

System parameters

The benchmark was executed on following system:

Laptop Acer TimelineX 5820TG
CPU:	Intel Core i5 M460 2C/4T @ 2.53 GHz
RAM:	4GB DDR3 1067 MHz
SSD:	Kingston SSDNow V300 128GB
OS:	Windows 10 (10.0.14393.105)

Little bit older hardware, I know, but the main point I am trying to show is to compare speed of various archive formats.

Large text file

First benchmark consisted of compressing one 109 MB text file. I used SQL dump from Wikipedia (skwiki-20160820-categorylinks.sql), which consist of mostly printable characters.

Here is the graph which I believe will provide answers to most questions:

Many various files

Second benchmark consisted of compressing directory with 86 subfolders and 6 131 files, total size 54.2 MB . I used FFMpeg source tree from 2 Sep 2016.

Linux image size

Third benchmark consisted of compressing 3.74 GB file with Linux filesystem, I used 2016-05-27-raspbian-jessie.img.

Results

• Clear winner is 7Z format (native for 7-Zip). Levels 1 and 3 will provide good compression quite fast.
• Use ZIP only when you have to transfer the files to systems where 7Z format is not an option. Otherwise, 7Z with lowest compression level will provide smaller file than ZIP with highest compression level, for 15x-70x less time. Amazing.
• Difference between 7Z compression levels 7 and 9 is almost non-existing. The result archive sizes were the same in all 3 cases, with run times difference on edge of significance.
• Similarly, difference in size between ZIP compression level 1 and 3 is none, difference in time is almost none.

Note for graphs: Please watch X and Y axis, they are not always linear and they don't necessary start at 0. I promise I will provide nicer graphs next time.

Introduction of Benchroom

To all the developers, coders, project managers, sysadmins and geeks: I am happy to announce the birth of Benchroom!
(Sorry for the that, I just had to)

For quite a long time I have been thinking about a platform for benchmarking various software. Questions like which compression algorithm has best run time / archive size ratio or which video codec is the fastest to be encoded in has kept me awake during long nights. Ok, to be honest, they didn't kept me awake, but I was still curious and many time it would make my decision during development a lot easier if I just had the right numbers.

For some of my questions I was able to find some partial benchmarks online, but I am quite a perfectionist and the number of various combinations in the benchmarks was not sufficient for me. Of course I don't blame the authors, they had done very good job, they probably run it all by hand, but I just wanted something more.

So, how to do more benchmarks without spending huge amount of time doing it? Easy, make it automatic! How? Spend ridiculous amount of time developing software which would do it automatic! So I started thinking and coding...

Right now, I have pre-alpha version of the benchmarking platform (which I called Benchroom) on GitHub (https://github.com/milan-fabian/benchroom). Any suggestions are of course welcome, but please be aware that it is really in pre-alpha state. If there will be some free time in next couple of weeks, I will post some ideas how I picture myself the entire platform.

Right now I am stress testing my older laptop with whole benchmark suite of 7-zip - first complete run of Benchroom. I have no idea how long it would take, but if the test goes well (keep you fingers crossed), I will post results here, so stay tuned.