Cromfs changelog cromfs - Copyright (C) 1992,2011 Bisqwit (http://iki.fi/bisqwit/) License: GPL3 Homepage: http://bisqwit.iki.fi/source/cromfs.html 1.5.10 2011-07-29 Joel Yliluoma - Update FSBAllocator version (GCC 4.6 compatibility; thanks Juha Nieminen) - Compilation fixes for GCC 4.6 - Added configure option: --enable debug - Fixed a bug that caused mkcromfs to crash when using large block sizes - Fixed a bug that caused mkcromfs to crash when given a file name rather than a directory name (thanks David Flater) - Fixed a bug that caused mkcromfs to malfunction in certain situations - Documentation updates (thanks David Flater) - Improve chances of successful compilation on a WIN32 platform 1.5.9.1 2010-12-21 Joel Yliluoma - Update LZMA version ("minor fixes") - Compilation fixes for GCC 4.5 - Compilation fixes for glibc 2.12.2 (thanks Viric) 1.5.9 2009-12-17 Joel Yliluoma - Update LZMA version (no relevant changes) - Add a few new gcc options when supported: -fvpt -ffunction-cse -ffunction-sections -fdata-sections -Xlinker --gc-sections - Eliminate some compiler warnings 1.5.8.9 2009-07-31 Joel Yliluoma - Compilation fixes for GCC 4.4 (thanks Gaetan Bisson) 1.5.8.8 2009-07-21 Joel Yliluoma - Compilation fixes for GCC 4.4 (thanks Gaetan Bisson) 1.5.8.7 2009-04-27 Joel Yliluoma - Compilation fixes for GCC 4.4 1.5.8.6 2009-03-26 Joel Yliluoma - Compilation fixes in 32-bit platforms and GCC-4.3.2 (thanks Gaetan Bisson) 1.5.8.5 2009-03-23 Joel Yliluoma - Added support for lutimes(), which makes it possible to change the timestamps of symlinks (unmkcromfs) - Further improved the memory efficiency of mkcromfs - Some minor code refactoring 1.5.8.4 2009-03-19 Joel Yliluoma - Fixed a crash bug introduced to mkcromfs in version 1.5.8, commit 3401942402870ded3eae2251dd029702dc3fc78b. - mkcromfs: Changed the default value of the -r option to 20000. 1.5.8.3 2009-03-14 Joel Yliluoma - Fixed a crash bug introduced in 1.5.8.1 (happens on ICC) 1.5.8.2 2009-03-14 Joel Yliluoma - More compilation fixes (thanks Gaetan Bisson) 1.5.8.1 2009-03-14 Joel Yliluoma - Compilation fixes in 32-bit platforms and GCC-4.3.2 (thanks Gaetan Bisson) 1.5.8 2009-03-13 Joel Yliluoma - Fixed a bug that caused vsnprintf not to be used even if it existed. - Fixed a random crashing bug (race condition) in unmkcromfs (it was introduced in version 1.5.6, commit 274443a59d977d021b24ca10c0b1d81f4d4d6b7b). - Added possibility to disable AutoIndex alltogether, cutting memory usage - After a lot of experimentation, added options for cutting down the memory usage of the block index in mkcromfs significantly. See below. - New mkcromfs option: --blockindexmethod (see documentation) - More optimized LZO compression for the blockindex - Added some more documentation in doc/ - Removed some unused source code files - Rewrote the fblock index from scratch, hopefully resulting in slightly more optimal refactored code - Detected a memory leak problem in ICC's implementation of OpenMP tasks... As a result, now not using OpenMP tasks anymore. - Fixed another memory leak problem in mkcromfs - Improved mkcromfs performance by refactoring some LZMA access code - Improved the speed of the hashing algorithm by some 30% (neglible impact to hash efficiency) - Added some sanity checks in unmkcromfs - Deprecated mkcromfs option: --blockifybufferlength - Deprecated mkcromfs option: --blockifybufferincrement - Deprecated mkcromfs option: --blockifyoptimizemethod - Updated LZMA to version 4.65. It includes a minor thread-related fix in LZMA encoder that has no practical consequence. 1.5.7 2009-02-01 Joel Yliluoma - Added support for OpenMP 3.0 based task constructs - Added protection against stupid umasks in unmkcromfs - Updated LZMA to version 4.63. It includes apparently nothing noteworthy. - Added some missing documentation files - Fixed some compilation problems - Added diagnosis for misapplied --16bitblocknums and --24bitblocknums - Changed the block index to a memory based system again, reverting the change introduced in version 1.5.5; however, with LZO compression, hopefully bringing down the RAM requirements by a bit or two without significantly affecting the mkcromfs running time adversely. 1.5.6.2 2008-12-16 Joel Yliluoma - Fixed the problem with AutoIndex&RealIndex and $TEMP filesystems unable to cope with > 2GB sparse files (Thanks Josh Rabinowitz) - Added the O_LARGEFILE flag to AutoIndex&RealIndex file opens. 1.5.6.1 2008-12-16 Joel Yliluoma - Added error resolution for disk write errors (such as low diskspace) (Thanks Petr Pudlak.) 1.5.6 2008-12-16 Joel Yliluoma - Added parallelization to block index search - Added parallelization to the directory scanning phase - Added predictive autoindex: now automatically indexes a block that immediately follows the point where a full overlap was found. This speeds up the cases where a multiple block long full overlap is found. - Changed unmkcromfs's output to be thread-safe with faux termcap control. - Fixed a bug that caused lots of almost empty fblocks to be created when --bruteforcelimit (-c) is set to 1 and --minfreespace (-s) is greater than 0 (default is 16). 1.5.5.4 2008-12-14 Joel Yliluoma - ./configure now detects systems where cromfs-driver-static cannot be built, such as Fedora 10. In those cases, the staticly linked program is silently dropped from the compilation chain. (Thanks to Josh Rabinowitz for making me aware of this problem.) - Minor changes to fix compilation using ICC. 1.5.5.3 2008-12-13 Joel Yliluoma - Compilation fixes on IA32 platform - Unified the output of "overlap fully" with and without autoindex 1.5.5.2 2008-12-13 Joel Yliluoma - Addressed some (stupid) compiler complaints. GCC's illadvised misinterpretation of Scott Meyer's Effective C++ rule 12 be damned. In the same breath, it appears that GCC does not honor -Wno-non-virtual-dtor. So we still get lots of warnings about non-virtual destructors. - Fixed a serious false positive bug in backwards_match_len -- this caused file system corruptions in mkcromfs since version 1.5.4.2. All users of versions 1.5.4.2, 1.5.5 or 1.5.5.1 are urged upgrade. 1.5.5.1 2008-12-09 Joel Yliluoma - Minor fixes - Publicized the git repository at git://bisqwit.iki.fi/cromfs.git 1.5.5 2008-12-08 Joel Yliluoma - Using Intel ICC as the default compiler when available for increased performance (powerful vectorization and automatic parallelisation). (Version 10 or greater recommended.) - mkcromfs: Changed the block index into a sparse temporary file to turn a memory bottleneck into a diskspace bottleneck, hopefully improving performance in huge filesystem creation (namely, prevent thrashing). Unfortunately, it will decrease performance in compressing of small filesystems. - Updated LZMA to version 4.62. It includes apparently "speed optimizations" and "some fixes". 1.5.4.2 2008-12-08 Joel Yliluoma - Somehow the 1.5.4.1 release did not have all of the changes I wanted to publish in it; 1.5.4.2 hopefully repacks it and does work on gcc version 4.3. 1.5.4.1 2008-11-24 Joel Yliluoma - The next release WIP has been long overdue because of hairy thread-related crashing problems, but for now, I'll release this bugfix release which fixes compilation problems on newer distributions and gcc versions. 1.5.4 2007-10-15 Joel Yliluoma - mkcromfs: Added --nosortbyfilename option. - mkcromfs: Added the --bsize_for option, and updated the filesystem format to allow each inode to have a different block size. Useful when creating filesystems that contain different types of files. Files of different type can have different block sizes. - Fixed some warnings spewed by gcc (none important). LZMA code still outputs dozens of those, but there's little I can do. 1.5.3.2 2007-09-04 Joel Yliluoma - Added include for sys/statvfs.h, fixing a compilation error relating to missing struct definition (thanks Thomas Munn). 1.5.3.1 2007-09-01 Joel Yliluoma - Oops! I forgot to include the ./configure in the package. 1.5.3 2007-08-29 Joel Yliluoma - mkcromfs: Added a feature to finish previously interrupted mkcromfs work (only if it was interrupted during the last stage (fblock LZMA compression or writing of fblocks into the result file)). - mkcromfs: Optimized the Boyer-Moore search algorithms, and added a Turbo Boyer-Moore search (now default) which should yield slightly better performance. - Added optional parallerism (threads) in cromfs-driver through OpenMP (requires gcc 4.2 and -fopenmp). See Makefile.sets - Also changed partially the thread systems in mkcromfs and unmkcromfs to OpenMP for simplicity of code. - Added a ./configure script (not autotools though). - Fixed an error in cromfs-driver that actually caused pessimization of performance in case of long file reads. - Since I could not find a single test case where the BWT or MTF filters yielded a compression improvement, I designed to remove them from this release to simplify the code. 1.5.2 2007-08-17 Joel Yliluoma - Added buffering in datasource, improving mkcromfs performance when the files are read from a slow filesystem (such as another cromfs mount) - Fixed error where every fblock would be mmapped at the same time whenever a new one must be started, overriding the bruteforcelimit setting. Thanks to "symmo" for reporting the error. - Fixed a bug that caused cromfs-driver to disregard the limits for cache maximum size, thereby consuming way more memory than it should. Thanks to "symmo" for reporting the error. - Added memory usage debugging in cromfs-driver. Invoke cromfs-driver with the -f option, and stat the mountpoint to make the driver output statistics. 1.5.1 2007-08-13 Joel Yliluoma - mkfromfs: The memory efficiency of the block index was further improved, enabling the use of a slightly shorter autoindex period if one so chooses. - mkcromfs, unmkcromfs: Improved threading performance by introducing a concept of permanent worker threads, instead of threads deleted and recreated at every request. - mkcromfs: Added --overlapgranularity (-g) option. It is experimental, and in theory improves reuse rate on periodical data. It restricts partial overlap into numbers that are evenly divisible with the given value. 1.5.0 2007-07-29 Joel Yliluoma - License change: Now it is under GPL3. (Previously GPL2.) - mkcromfs: Changed the order in which inode numbers are assigned, improving the access times, reducing directory fragmentation and making the recently added --dirparseorder a more useful option. It seems to also improve compression. - Internal changes: In an effort to improve portability of the code, moved most of the file access code into include files that handle sparseness, memory mapping and regular file reading/writing in one place. - mkcromfs: In attempt to improve the block scheduler, a new --blockifyoptimizemethod value was introduced. It uses a variation of the Asymmetric Travelling Salesman Problem to determine an optimal order for blockifying. It may or may not improve compression, but it does consume a lot more CPU time. - mkcromfs: The memory efficiency of the block index was improved, enabling the use of slightly larger autoindex ratio if one so chooses. (Note: The actual effect of the change is yet untested.) - mkcromfs: Fixed a bug in autoindex: Index hits did not assign a block number to the index. Compression power improved again. - General: Using posix_fadvice() to hint the kernel about proper page cache strategy. Theoretically should improve performance. - Experimented with asynchronous IO (AIO) in unmkcromfs. However, it seemed to not do much good, so it is now disabled by default, enableable only by a #define. - cromfs-driver: Changed the cache so that now oldest accessed flocks are deallocated first, instead of randomly choosing one. Same for directory listings. Should improve performance. - mkcromfs: Added threading to improve performance on multicore systems. (New option: --threads). - unmkcromfs: Added threading to improve performance on multicore systems. (New option: --threads). - mkcromfs: Raised the default value of --bruteforcelimit to 2, from the previous value of 1. (0 and 1 are equal.) - Added -ldl in Makefile to enable linking with fuse 2.7. - mkcromfs: Replaced --autoindexratio (-a) with --autoindexperiod (-A) that has inverse, but cleaner, semantics. The old option will still work, but is now deprecated. - mkcromfs: Improved autoindexer performance. - unmkcromfs: Fixed an error that prevented hardlinked files from being extracted properly in some cases. - Documentation updates (some new compression tests & comparisons) 1.4.1 2007-07-15 Joel Yliluoma - Added autochoosing heuristics for the -3 (--24bitblocknums), -2 (--16bitblocknums) and -k (--packedblocks) options, because it can improve compression and those who profile mkcromfs, will do so with its default options :) - Added --32bitblocknums (-4) for overriding the autochoosing heuristics. - Replaced the -k (--packedblocks) option with --nopackedblocks for overriding the heuristics. - Fixed a bug that caused almost-empty fblocks to be created needlessly. - Fixed a bug that prevented --lzmabits from being parsed properly. - Reorganized source code directories, resulting in a larger diff than normally. - Experimentally added BWT and MTF filters. Both seem to do more harm than good, so they're disabled by default. Apparently LZMA benefits from neither. - Fixed a severe bug introduced in 1.4.0 that prevented older filesystems from being mounted properly. 1.4.0 2007-07-13 Joel Yliluoma - Removed filename length limit - Rewrote most of mkcromfs. A new algorithm of organizing the blocks of files before locating them in fblocks ("blockifying") should improve compression. This does not change the actual format of the filesystem. - Added several blockify-related optimization options in mkcromfs. (--blockifybufferlength, --blockifybufferincrement, and --blockifyoptimizemethod). - Speed improvements in cromfs-driver: - Optimized read_file_data() and read_dir() in case of heavily fragmented filesystems. It will, however, incur a slight performance penalty for non-fragmented filesystems. - Added an implementation of dir_lookup(), significantly improving the performance of lookups especially on heavily fragmented filesystems. - Added -3 (--24bitblocknums), -2 (--16bitblocknums) and -k (--packedblocks) options in mkcromfs (extension to cromfs file format). Each of these options have some potential for improving the compression. Even if LZMA negates it, at least it reduces memory traffic during cromfs-driver operation, so it is recommended to use those where possible. - Fixed a few endianess problems. - Added --lzmafastbytes option in mkcromfs and cvcromfs, making the number of LZMA "fast bytes" configurable. Also raised the default from 64 to 273 (maximum). Larger number usually results in a better compression. - Added --lzmabits option in mkcromfs and cvcromfs for LZMA compression finetuning. - Added possibility to alter the order in which different types of files are compressed. (Options --blockifyorder and --dirparseorder in mkcromfs.) - Upgraded LZMA SDK from version 4.43 to 4.49 beta. 1.3.0 2007-06-30 Joel Yliluoma - New format is under work (filesystem write-mounting) - cromfs-driver now goes background (fork) by default (use -f to go foreground) - Added -v (--verbose) in unmkcromfs (and tuned down verbosiness by default) - Documented how to use autofs with cromfs (thanks Petr Pudlak) 1.2.5 2007-02-14 Joel Yliluoma - Tentatively fixed endianess problems (needs testing!) - Updated box art - Fixed a readdir problem that caused GNU find to not work properly (because it relied on undocumented d_type field in readdir results). - Only cromfs-driver is affected; filesystems do not need to be reconstructed. 1.2.4.5 2007-02-12 Joel Yliluoma - Documentation updates 1.2.4.4 2007-01-27 Joel Yliluoma - Fixed yet another compilation problem, by reverting to fuse 2.5 API - Documentation updates 1.2.4.3 2007-01-26 Joel Yliluoma - Fixed another compilation problem 1.2.4.2 2007-01-11 Joel Yliluoma - Fixed a compilation problem in unmkcromfs 1.2.4.1 2006-12-05 Joel Yliluoma - Added -l (--followsymlinks) in mkcromfs - Made --simgraph in unmkcromfs a lot faster 1.2.4 2006-12-04 Joel Yliluoma - Added --simgraph in unmkcromfs 1.2.3 2006-08-31 Joel Yliluoma - Added -x and -X (--exclude and --exclude-from) in mkcromfs and unmkcromfs (thanks to Jeff Strunk for suggesting this) 1.2.2 2006-08-23 Joel Yliluoma - Added a static cromfs-driver (cromfs-driver-static) for bootdisks 1.2.1 2006-08-09 Joel Yliluoma - Added -q, -qq, -qqq in mkcromfs (thanks to Gilson Laurent for suggesting this) - Fixed the weird permissions of mountpoints - Disabled GCC option -ftree-vectorize by default, because GCC-4 is not widespread enough yet. - Large file support fixed in unmkcromfs - Fixed the stat()/lstat() calls; they previously reported blkcounts 8 times to small, causing "du" to report wrong sizes. 1.2.0 2006-06-06 Joel Yliluoma - New filesystem format (old filesystems are still supported) The inode table inode and root directory inode are now compressed. - New utility: cvcromfs (to convert filesystems between versions 01 and 02) (also helpful if new and more effecient versions of LZMA are released, to recompress existing filesystems) 1.1.7 2006-06-05 Joel Yliluoma - Hardlink extraction support in unmkcromfs - Added tests: make test 1.1.6.1 2006-06-04 Joel Yliluoma - Compilation workaround - More options in unmkcromfs 1.1.6 2006-06-04 Joel Yliluoma - Added a new utility: unmkcromfs It extracts the contents of a cromfs volume without mounting it using an optimized algorithm that minimizes duplicate data access. 1.1.5 2006-06-03 Joel Yliluoma - Made the --bruteforcelimit option (-c) more useful in mkcromfs (performance tuning). 1.1.4.3 2006-06-01 Joel Yliluoma - Minor tuning in mkcromfs - Nice image of a box to the program homepage. :) 1.1.4.2 2006-05-22 Joel Yliluoma - Documentation updates 1.1.4.1 2006-05-22 Joel Yliluoma - mkcromfs speed optimization: Bayer-Moore search algorithm (Noticeably faster than the Horspool version) - Fixed mkcromfs - mkcromfs from 1.1.4 crashed in certain situations - mkcromfs 1.1.4 failed to create a filesystem (a debugging option was accidentally left activated) 1.1.4 2006-05-21 Joel Yliluoma - mkcromfs speed optimization: Bayer-Moore-Horspool search algorithm - Documentation updates 1.1.3.1 2006-05-18 Joel Yliluoma - Documentation updates - Rewrite of cromfs::read_fblock to utilize mmap64 1.1.3 2006-05-17 Joel Yliluoma - Changing some occassions of std::map and std::multimap into gnu hash_map in mkcromfs. This might enable some extra speed. (Though the bottleneck is in the disk access.) - mmap-based optimizations in mkcromfs (this time implemented in a way that shouldn't make reiser4fs crash.) - Possibly a crash bug fix in mkcromfs 1.1.2.3 2006-05-16 Joel Yliluoma - Upgraded LZMA SDK to version 4.42. (Ps: Someone _really_ should rewrite the LZMA compressor using _SANE_ C++. The official version is just horrible. It goes to very extremes to hack&introduce concepts which are foreign to C++, such as GUID, variants and C# style interfaces, and it reinvents many standard concepts such as integer types, smart pointers and streams, and as such, the code is very unmaintainable and likely suboptimal too. On the other hand, the C# version seems quite tidy... I wonder if it's the authoritative implementation? (hint: port it) I can only have nightmares trying to get into the mindscape of the programmer who wrote that horrible C++ code version.) - Documentation updates 1.1.2.2 2006-05-15 Joel Yliluoma - Tuned the --bruteforcelimit function - Documentation updates 1.1.2.1 2006-05-15 Joel Yliluoma - Fixed --bruteforcelimit. It didn't work in 1.1.2. - Documentation updates 1.1.2 2006-05-15 Joel Yliluoma - Ownerships are now saved too (in a mostly backward+forward compatible way). - Added new option in mkcromfs, --bruteforcelimit. 1.1.1 2006-05-14 Joel Yliluoma - Data integrity problem hopefully fixed now. - Improved the usefulness of the caching in read_dir. 1.1.0.2 2006-05-14 Joel Yliluoma - Workaround for some noticed data integrity problem (some compressed blocks were assumed as uncompressed, and compressed over again) 1.1.0.1 2006-05-14 Joel Yliluoma - A compilation fix (thanks to Cu Van Chuoi) 1.1.0 2006-05-14 Joel Yliluoma - Removed OpenSSL dependency - Replaced MD5 with CRC32 for smaller memory usage in mkcromfs (This has no effect in data integrity; a hash is just a hash.) - Worked around an instability problem when used together with reiserfs4 (mmapping a sparse file as read-write with copy-on-write mapping, and changing that underlying file, seems to under some circumstances cause kernel crash and kernel BUGs when the underlying filesystem is reiser4.) - Disabled the auto deallocator hack for now; it hasn't worked properly. - Changed FSIZE semantics: fsize now denotes the maximum uncompressed fblock size, not the maximum compressed fblock size. This made mkcromfs somewhat easier in design, and probably more useful for the users too. - Removed mkcromfs option: --tryhard - Added mkcromfs option: --autoindexratio - Documentation updates 1.0.6 2006-05-12 Joel Yliluoma - Another compilation fix - Documentation updates - mkcromfs's temporary directory now configurable (recognizes the environment variable TEMP or TMP) - mkcromfs optimization: Uses mmapping to compare portions of files. Most effective when used together with --decompresslookups. 1.0.5.1 2006-05-12 Joel Yliluoma - A compilation fix 1.0.5 2006-05-11 Joel Yliluoma - Added compatibility for older versions of Fuse (2.5.2 now supported, previously only compiled on 2.6.0) - Changed filesystem format: FBLOCKs are now stored without padding. This will result in better compression, and opens opportunities to make mkcromfs faster. - Added commandline option in mkcromfs: --tryhard 1.0.4 2006-05-11 Joel Yliluoma - Documentation updates - Fixed mkcromfs error from 1.0.3 version - Added commandline options in mkcromfs: --decompresslookups, --randomcompressperiod and --minfreespace 1.0.3 2006-05-11 Joel Yliluoma - Documentation updates - Broken mkcromfs release - Added statistics reporting in mkcromfs at end 1.0.2 Joel Yliluoma - Added support for block devices and character devices - Documentation updates - Added commandline options in mkcromfs: --bsize, --fsize - Fixed exception handling in cromfs-driver - Changed cromfs-driver so that uid/gid now refer to the one who mounted the filesystem, not root - Disabled the tracing output in cromfs-driver by default - Changed attribute cache timeout to ~infinite - Added periodical deallocation of BLKTAB - Added cache of read_dir() when it's used by lookup() - Misc tuning in mkcromfs - mkcromfs now can stand running multiple instances at same time - Changed the LZMA dictionary size selection method (heavier settings now) 1.0.1 Joel Yliluoma - Changed the LZMA implementation slightly more straight-forward - Changed filesystem format: BLKTAB is now compressed within BLKDATA - A slight speed optimization (later to become the "decompresslookups" option) 1.0.0 Joel Yliluoma - Initial release