Comments for The Pseudo Random Bit Bucket

Comment on Reducing OpenSolaris ramdisk greed by moinakg

moinakg — Sat, 18 May 2013 12:32:25 +0000

Thanks. I am using one of the predefined WordPress themes. Will check it out.

Comment on Reducing OpenSolaris ramdisk greed by 83597

83597 — Sat, 18 May 2013 12:23:07 +0000

Simply wanted to let you know that I found your blog on and while
I appreciated checking out your post, it appears your blog
acts up in a couple browsers. If I use Firefox, it comes up okay, but
if I use Chrome, it comes up appearing overlapped and off-kilter.

Just so you know.

Comment on Hadoop – too much hype by moinakg

moinakg — Sun, 12 May 2013 17:50:46 +0000

Yes I agree that the HotSpot JIT does a darn well job, though it can require tweaking via the code cache size parameter. However my big beef is with the GC piece. I see a significant effort being directed to handle GC related issues in most Java software R&D leading eventually to fancy off-heap storage for big datasets which looks like new/delete in disguise. I have the odd wish of seeing a Java variant with explicit memory management, no GC at all. The VM could be made a lot more lightweight in that case.

Comment on Hadoop – too much hype by Jean-Francois Im (@jeanfrancoisim)

Jean-Francois Im (@jeanfrancoisim) — Sun, 12 May 2013 16:44:00 +0000

It’s certainly possible to write relatively high performance on the JVM; just because there is a crapton of poor Java developers doesn’t mean that the technology itself sucks. It’s not perfect (I don’t think it does SIMD ops yet on x86/x64) but it certainly does allow one to write reasonably fast code in a fraction of the time it would take to do so in C/C++ or assembly (for example, I added clustering and multi-node distributed processing to an app in a short afternoon on the JVM, which would’ve been a major pain in C++ — even with 0MQ).

As we both agree, though, people who don’t understand how the computer works underneath and all the layers in between will write crap code that’s suboptimal.

Comment on Pcompress 2.1 released with fixes and performance enhancements by moinakg

moinakg — Sun, 12 May 2013 06:09:43 +0000

Thanks for letting me know. This looks like a bug in an invalid input handling routine! ‘Adapt2′ mode uses libbsc which can take a max segment size of 1024m. Obviously ‘-s 1280m’ exceeds that. But then the ‘Floating point exception’ looks like a simple division by zero bug while printing the statistics.

Comment on Pcompress 2.1 released with fixes and performance enhancements by Slowpoke

Slowpoke — Sun, 12 May 2013 01:11:36 +0000

Thanks for this nice and usefull program!
Did a quick test on Ubuntu 12.10, pcompress crashed with these switches If need be you can get the archive here: http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Automated%20Builds/

% pcompress -E -D -L -P -B 1 -M -C -c adapt2 -l 14 -s 1280m mingw-w64-bin_x86_64-linux_20130505.tar
Scaling to 1 thread
Max allowed chunk size for LIBBSC is: 1073741824
Error compressing file: mingw-w64-bin_x86_64-linux_20130505.tar

Compression Statistics
======================
Total chunks : 0
Best compressed chunk : 0 B(0.00%)
Worst compressed chunk : 0 B(0.00%)
Floating point exception (core dumped)

(Linux 3.5.0-28-generic #48-Ubuntu SMP Tue Apr 23 23:03:38 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux, gcc (Ubuntu/Linaro 4.7.2-2ubuntu1) 4.7.2, Core-i5/16GB RAM)

Comment on About by moinakg

moinakg — Tue, 07 May 2013 17:49:37 +0000

Thanks. Though Pcompress is far from being an FS actually a few of the core ideas can be leveraged in existing filesystems like Btrfs. For example the scalable segmented deduplication idea can be leveraged in the volume layer for offline dedupe. The variable-block sliding-window chunking I am using has performance close to fixed-block chunking so it can be used as well among other things.
So one is welcome to leverage stuff for Btrfs if it helps. I could help anyone willing to do that. I have intentions to evolve Pcompress into a deduplicated archival/object storage appliance with the features sitting within the lower level block I/O layer with some existing filesystem on the top.

Comment on About by Bgm0

Bgm0 — Tue, 07 May 2013 16:52:53 +0000

You know Pcompress is soo awesome that it could become a FS . Btrfs is trying to include features present in Pcompress. How about a merge of the two ?

Comment on Vectorizing xxHash for Fun and Profit by sanmayce

sanmayce — Tue, 23 Apr 2013 18:10:05 +0000

Hi again moinakg, I am happy to share my first XMM attempt: FNV1A_YoshimitsuTRIADiiXMM: Wanted to see how SIMDed main loop would look like: One of the main goals: to stress 128bit registers only and nothing else, for now 6 in total, in fact Intel uses the all 8. Current approach: instead of rotating the 5 bits within the DWORD quadruplets I chose to do it within the entire DQWORD i.e. XMMWORD. // #define xmmloadu(p) _mm_loadu_si128((__m128i const*)(p)) // #define _rotl_KAZE128(x, n) _mm_or_si128(_mm_slli_si128(x, n) , _mm_srli_si128(x, 128-n)) // uint32_t FNV1A_Hash_YoshimitsuTRIADiiXMM(const char *str, uint32_t wrdlen) // { // const char *p = str; // ... // if (wrdlen >= 4*24) { // Actually 4*24 is the minimum and not useful, 200++ makes more sense. // Loop_Counter = (wrdlen/(4*24)); // Loop_Counter++; // Second_Line_Offset = wrdlen-(Loop_Counter)*(4*3*4); // for(; Loop_Counter; Loop_Counter--, p += 4*3*sizeof(uint32_t)) { // xmm0 = xmmloadu(p+0*16); // xmm1 = xmmloadu(p+0*16+Second_Line_Offset); // xmm2 = xmmloadu(p+1*16); // xmm3 = xmmloadu(p+1*16+Second_Line_Offset); // xmm4 = xmmloadu(p+2*16); // xmm5 = xmmloadu(p+2*16+Second_Line_Offset); // hash32xmm = _mm_mullo_epi32(_mm_xor_si128(hash32xmm , _mm_xor_si128(_rotl_KAZE128(xmm0,5) , xmm1)) , PRIMExmm); // hash32Bxmm = _mm_mullo_epi32(_mm_xor_si128(hash32Bxmm , _mm_xor_si128(_rotl_KAZE128(xmm3,5) , xmm2)) , PRIMExmm); // hash32Cxmm = _mm_mullo_epi32(_mm_xor_si128(hash32Cxmm , _mm_xor_si128(_rotl_KAZE128(xmm4,5) , xmm5)) , PRIMExmm); // } // // The simplest mumbo-jumbo mix: // hash32xmm = _mm_mullo_epi32(_mm_xor_si128(hash32xmm , _rotl_KAZE128(hash32Bxmm,5)) , PRIMExmm); // hash32xmm = _mm_mullo_epi32(_mm_xor_si128(hash32xmm , _rotl_KAZE128(hash32Cxmm,5)) , PRIMExmm); // hash32 = (hash32 ^ _rotl_KAZE(hash32xmm.m128i_u32[0],5) ) * PRIME; // hash32 = (hash32 ^ _rotl_KAZE(hash32xmm.m128i_u32[1],5) ) * PRIME; // hash32 = (hash32 ^ _rotl_KAZE(hash32xmm.m128i_u32[2],5) ) * PRIME; // hash32 = (hash32 ^ _rotl_KAZE(hash32xmm.m128i_u32[3],5) ) * PRIME; // return hash32 ^ (hash32 >> 16); // } else if (wrdlen >= 24) { // ... // } /* ; mark_description "Intel(R) C++ Compiler XE for applications running on IA-32, Version 12.1.1.258 Build 20111011"; ; mark_description "-Ox -TcHASH_linearspeed_FURY.c -FaHASH_linearspeed_FURY_Intel_IA-32_12 -FA"; .B4.4: lea edi, DWORD PTR [esi+esi*2] inc esi shl edi, 4 cmp esi, edx movdqu xmm7, XMMWORD PTR [ecx+edi] movdqu xmm6, XMMWORD PTR [16+ebx+edi] movdqu xmm5, XMMWORD PTR [32+ecx+edi] movdqa xmm1, xmm7 pslldq xmm1, 5 psrldq xmm7, 123 por xmm1, xmm7 movdqu xmm7, XMMWORD PTR [ebx+edi] pxor xmm1, xmm7 pxor xmm2, xmm1 movdqa xmm1, xmm6 pslldq xmm1, 5 psrldq xmm6, 123 por xmm1, xmm6 movdqu xmm6, XMMWORD PTR [16+ecx+edi] pxor xmm1, xmm6 movdqa xmm6, xmm5 pslldq xmm6, 5 pxor xmm3, xmm1 psrldq xmm5, 123 por xmm6, xmm5 movdqu xmm5, XMMWORD PTR [32+ebx+edi] pxor xmm6, xmm5 pxor xmm4, xmm6 pmulld xmm2, xmm0 pmulld xmm3, xmm0 pmulld xmm4, xmm0 jb .B4.4 I wrote YoYo r.1+, my [CR]LF lines hasher in order to see what dispersion it has, in the next dump it hashed 1,048,576 Knight Tours using 20bit hash table: YoYo - [CR]LF lines hasher, r.1+ copyleft Kaze. Note1: Incoming textual file can exceed 4GB and lines can be up to 1048576 chars long. Note2: FNV1A_YoshimitsuTRIADiiXMM needs SSE4.1, so if not present YoYo will crash. Polynomial(s) used: CRC32C2_8slice: 0x8F6E37A0 HashSizeInBits = 20 Allocating KEY memory 1024KB ... OK Allocating HASH memory 4MB ... OK Allocating HASH memory 4MB ... OK Allocating HASH memory 4MB ... OK Hashing all the LF ending lines encountered in 136,314,880 bytes long file ... Keys vs Slots ratio: 1:1 or 1,048,576:1,048,576 FNV1A_YoshimitsuTRIADiiXMM : Keys = 00,000,000,000,001,048,576; 000,000,004 x MAXcollisionsAtSomeSlots = 0,000,000,010; HASHfreeSLOTS = 0,000,413,289; HashUtilization = 060%; Collisions = 0,000,413,289 FNV1A_YoshimitsuTRIADii : Keys = 00,000,000,000,001,048,576; 000,000,002 x MAXcollisionsAtSomeSlots = 0,000,000,009; HASHfreeSLOTS = 0,000,385,367; HashUtilization = 063%; Collisions = 0,000,385,367 CRC32C2_8slice : Keys = 00,000,000,000,001,048,576; 000,000,001 x MAXcollisionsAtSomeSlots = 0,000,000,009; HASHfreeSLOTS = 0,000,385,451; HashUtilization = 063%; Collisions = 0,000,385,451 Physical Lines: 1,048,576 Shortest Line : 128 Longest Line : 128 Obviously the function needs some tuning, sadly my CPU T7500 supports up to SSSE3 and I cannot play with it (Q9550S was used for above dump). In my view FNV1A_YoshimitsuTRIADiiXMM is the fastest (regarding bandwidth) hasher, I see it dethroned only by incoming FNV1A_JAMIROQUAI (aka FNV1A_YoshimitsuTRIADiiYMM). http://en.wikipedia.org/wiki/File:Jamiroquai_2.jpg I won't disturb you until I buy AVX machine and write 'JAMIROQUAI'. Regards

Comment on Vectorizing xxHash for Fun and Profit by moinakg

moinakg — Thu, 18 Apr 2013 14:27:36 +0000

That is an awesome resul t. In the current xxHash code there are only two levels of interleaving leaving enough XMM registers to do two more levels of interleaving which should improve speed much more. However there was also the question of retaining hash value compatibility with non-XMM code. Having more levels of interleaving will actually slow down the non-XMM loop on CPUs that do not have SSE 4.1 as the number of intermediate variables will overflow the available registers.