This project is read-only.

Using LZ4 Stream, TCP?

Aug 19, 2013 at 2:26 PM
Edited Aug 19, 2013 at 2:55 PM
I don´t understand how i am supposed to use the LZ4 Stream?

If i want to do something like this:
tcp.GetStream().Write(BitConverter.GetBytes(bsize), 0, intsize);
tcp.GetStream().Write(ms.GetBuffer(), 0, (int)bsize);
But with LZ4, how am i supposed to do it?
Or is it not that kind of a stream?

Cause i am guessing i am supposed to do like:
var strum = new LZ4Stream(tcp.GetStream(), System.IO.Compression.CompressionMode.Compress, false);
and then just change tcp.GetStream() to strum, but it doesn´t seem to work as expected.

EDIT:

Well i get the error that multiple chunk passes are not supported.

Not sure why though, i get it here:
int currentPosition = 0;
 while (currentPosition < length)
{
currentPosition += strum.Read(tempBytes, currentPosition, length - currentPosition);
//currentPosition += tt1.GetStream().Read(tempBytes, currentPosition, length - currentPosition);
}
EDIT 2:

Got it to work, needed to change the blocksize, why is that?
changed to 1024 for both decompression and compression.
Aug 19, 2013 at 3:00 PM
I'm not sure what you are trying to do.
Can you send me mini-solution showing what you want to achieve?
Aug 19, 2013 at 3:09 PM
I am trying to send stuff in a loop, and recieve them.
It works with normal tcp streaming.

And it seems to work with LZ4 Stream, But not when i send .Bmp (i send files). I guess they are to big, or something.
Cause if i increase the BlockSize to high, i get exceptions, and if i have it to low, it stutter and lags.

But here is the sending part:
   while (tcp.Connected && capcon == true)
                            {
                                    using (var ms = new MemoryStream())
                                    {
                                        FastMethods.CaptureBitBit(hwnd, JpegParam, jpeg, ms);

                                        bsize = ms.Length;

                                        strum.Write(BitConverter.GetBytes(bsize), 0, intsize);

                                        //tcp.GetStream().Write(BitConverter.GetBytes(bsize), 0, intsize);

                                        strum.Write(ms.GetBuffer(), 0, (int)bsize);
                                        //tcp.GetStream().Write(ms.GetBuffer(), 0, (int)bsize);
                                    }
                                }
So FastMethods is just a capturing thing, and saves a bitmap into a memorystream (i want to use .bmp, not much use to compress jpeg).

And then i send the size of the data, and then the actual data.

Then on the receiving side:
//tt1.GetStream().Read(lenArray, 0, 4);
strum.Read(lenArray, 0, 4);

while(tt1.connected){
 tempBytes = new byte[length];
                                    ms = new MemoryStream(tempBytes);
                                    int currentPosition = 0;
                                    while (currentPosition < length)
                                    {
                                        currentPosition += strum.Read(tempBytes, currentPosition, length - currentPosition);
                                        //currentPosition += tt1.GetStream().Read(tempBytes, currentPosition, length - currentPosition);
                                    }
   newImage = Image.FromStream(ms);

    //tt1.GetStream().Read(lenArray, 0, 4);

  strum.Read(lenArray, 0, 4);

      length = BitConverter.ToInt32(lenArray, 0);

 }
So i receive the size from outside the loop for the first run, then i receive it at the end.

And in the other loop, i get the data, then after that i can get the image itself.

Sorry for the formatting, i always use Auto Formatting.
Aug 19, 2013 at 3:11 PM
Edited Aug 19, 2013 at 3:15 PM
LZ4Stream uses BLOCK compression, so you need to Close/Dispose it to flush last chuck of data.
using (var lz4Stream = new LZ4Stream(tcpStream, CompressionMode.Compress))
{
    // write data...
}
You don't need to change block size, in most cases default value is just fine (the default is 1MB, and every block adds some overhead, and cripples compression ratio - setting it to 1024 bytes - bad idea)
Aug 19, 2013 at 3:14 PM
As I said, I can't see Using (or Close or Dispose) when you're writing to LZ4Stream.
Aug 19, 2013 at 3:23 PM
Edited Aug 19, 2013 at 3:30 PM
Wait, so i need to handle it differently compared to Tcp.GetStream()?
As, i do dispose of them, but only outside the loop, meaning, i don´t remake the Client on every loop, couldn´t even if i wanted to though, it would break the connection.

hmm, doesn´t seem to work, did like this:
   while (tcp.Connected && capcon == true)
                            {
                                try
                                {
                                    using (var ms = new MemoryStream())
                                    {
                                        using (var strum = new LZ4Stream(tcp.GetStream(), System.IO.Compression.CompressionMode.Compress, false))
                                        {
                                            FastMethods.CaptureBitBit(hwnd, JpegParam, jpeg, ms);

                                            //if (bsize != ms.Length && tcp.Connected)
                                            //{

                                            bsize = ms.Length;
                                            strum.Write(BitConverter.GetBytes(bsize), 0, intsize);
                                            //tcp.GetStream().Write(BitConverter.GetBytes(bsize), 0, intsize);
                                            strum.Write(ms.GetBuffer(), 0, (int)bsize);
                                        }
                                        //tcp.GetStream().Write(ms.GetBuffer(), 0, (int)bsize);
                                        //}
                                    }
                                }
Excuse the mess.
Aug 19, 2013 at 3:51 PM
Edited Aug 19, 2013 at 3:52 PM
You are giving me some partial code. I don't know where communication starts and where it ends.

Bottom line is: LZ4Stream needs to be closed because it is IDisposable, and it relias on IDisposable being used properly.
When you create LZ4Stream you need to dispose it somewhere. In you example you are disposing it after sending first picture. Do you want this? I doubt it.

If you send me working example with TCP stream I can analyze this and send you "LZ4-ed" version of it.
Aug 19, 2013 at 3:55 PM
Yeah well, the rest is a total mess with nothing to really do with the actual communication.
If you look at TCP GetStream() i have commented out, those work perfectly.
And i am just trying to replace them with LZ4Stream.

And Now i understand what you mean, and yeah, i am keeping check with that.
Thought you meant i needed to dispose of it all the time.


But is you look at the GetStream, and then Strum, i have done exactly the same, shouldn´t it work, or am i missing something?
(If you ignore that i dispose of it all the time in the last example, which i am not doing now, i dispose of it outside the loop when i am done).
Aug 19, 2013 at 4:02 PM
You didn't. You showed me that in this lines you did the same. Not in whole code. Ther big picture is 'tcp' comes from somewhere, and something is done to it, before the code you sent, and after the code you sent.
It get's create and disposed (even implicitly), and LZ4Stream isn't.
Aug 19, 2013 at 4:08 PM
Edited Aug 19, 2013 at 4:11 PM
Okay well, will show more of the code and try to clean up the uneeded parts.

Send Part:
        private void Send()
        {
            try
            {
                using (var proc = Process.GetProcessesByName(Proccessname.Text)[0])
                {
                    hwnd = proc.MainWindowHandle;
                }
                using (EncoderParameters JpegParam = new EncoderParameters())
                {
                    JpegParam.Param[0] = new EncoderParameter(System.Drawing.Imaging.Encoder.Quality, 90L);

                    while (capcon == true)
                    {

                        using (var tcp = new TcpClient())
                        {
                            tcp.NoDelay = true;
                            try
                            {
                                if (!tcp.Connected)
                                    tcp.Connect(adress);
                            }
                            catch (Exception e)
                            {
                                if (e is SocketException || e is IOException) { }
                                else
                                    MessageBox.Show(e.Message + " Tcp Connect : Send");
                            }

                            using (var strum = new LZ4Stream(tcp.GetStream(), System.IO.Compression.CompressionMode.Compress, false))
                            {
                                while (tcp.Connected && capcon == true)
                                {
                                    try
                                    {
                                        using (var ms = new MemoryStream())
                                        {
                                            FastMethods.CaptureBitBit(hwnd, JpegParam, jpeg, ms);

                                            //if (bsize != ms.Length && tcp.Connected)
                                            //{

                                            bsize = ms.Length;
                                            strum.Write(BitConverter.GetBytes(bsize), 0, intsize);
                                            //tcp.GetStream().Write(BitConverter.GetBytes(bsize), 0, intsize);
                                            strum.Write(ms.GetBuffer(), 0, (int)bsize);
                                            //  tcp.GetStream().Write(ms.GetBuffer(), 0, (int)bsize);
                                        }

                                        //}
                                    }

                                    catch (Exception e)
                                    {
                                        if (e is SocketException || e is IOException)
                                        {
                                            break;
                                        }
                                        else
                                            MessageBox.Show(e.Message + ": Send Error");
                                    }
                                }
                            }
                        }
                    }

                }
            }
            catch (Exception e)
            {
                MessageBox.Show(e.Message + "SEND");
            }
        }
Receiving:
    private void Listen()
        {
            try
            {
                tcplisten.Start();
                using (tt1 = tcplisten.AcceptTcpClient())
                {
                    using (var strum = new LZ4Stream(tt1.GetStream(), System.IO.Compression.CompressionMode.Decompress, false))
                    {
                        tt1.NoDelay = true;
                        byte[] lenArray = new byte[intsize];
                        //tt1.GetStream().Read(lenArray, 0, 4);


                        strum.Read(lenArray, 0, 4);
                        
                        Int32 length = BitConverter.ToInt32(lenArray, 0);
                        var tempBytes = new byte[length];
                        var ms = new MemoryStream(tempBytes);
                        if (checkBox1.Checked && tt1.Connected)
                        {

                            try
                            {

                                SlimDX.Windows.MessagePump.Run(form, () =>
                                {
                                    if (checkBox1.Checked && tt1.Connected)
                                    {
                                        tempBytes = new byte[length];
                                        ms = new MemoryStream(tempBytes);

                                        int currentPosition = 0;
                                        while (currentPosition < length)
                                        {
                                            currentPosition += strum.Read(tempBytes, currentPosition, length - currentPosition);
                                            //currentPosition += tt1.GetStream().Read(tempBytes, currentPosition, length - currentPosition);
                                        }

                                 //       tt1.GetStream().Read(tempBytes, 0, length);

                                        //newImage = Image.FromStream(ms);

                                        strum.Read(lenArray, 0, 4);

                                        //tt1.GetStream().Read(lenArray, 0, 4);
                                        length = BitConverter.ToInt32(lenArray, 0);
                                    }
                                });
                            }
                            catch (Exception e)
                            {
                                if (e is ObjectDisposedException) { }
                                else if (e is IOException || e is ArgumentException || e is SocketException)
                                {
                                    return;
                                }
                                else
                                    MessageBox.Show(e.Message + "Listen() 2");
                            }
                        }
                    }
                }
            }
            catch (Exception e)
            {
                if (e is SocketException || e is IOException || e is OverflowException || e is InvalidOperationException)
                {
                    return;
                }
                else
                    MessageBox.Show(e.Message + "Listen()");
            }

        }
This is the whole code on the send and receiving, Receiving got a bit cleaned up, but it was just stuff like Device initialization.
Aug 20, 2013 at 12:31 AM
Ok.
So it did trace back to bug in LZ4Stream. It was working perfectly with stream which do provide all the data (like FileStream) but not so great with stream which deliver data over time (like NetworkStream).
I'll fix it when I'm back from holiday.

Although, I managed to make it work with additional pass-through stream.

Example server (no difference here):
using System;
using System.IO;
using System.IO.Compression;
using System.Net;
using System.Net.Sockets;

namespace LZ4.Test.Service
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            var folder = args[0];

            var listener = new TcpListener(IPAddress.Any, 4444);
            listener.Start();

            while (true)
            {
                try
                {
                    Console.WriteLine("Waiting for client...");
                    var client = listener.AcceptTcpClient();
                    using (var tcpStream = client.GetStream())
                    using (var lz4Stream = new LZ4Stream(tcpStream, CompressionMode.Compress))
                    using (var writer = new BinaryWriter(lz4Stream))
                    {
                        foreach (var file in Directory.GetFiles(folder, "*", SearchOption.AllDirectories))
                        {
                            Console.WriteLine(file);

                            writer.Write(file);
                            var bytes = File.ReadAllBytes(file);
                            writer.Write(bytes.Length);
                            writer.Write(bytes);
                        }

                        writer.Write(string.Empty);
                    }
                }
                catch (Exception e)
                {
                    Console.WriteLine("{0}: {1}", e.GetType().Name, e.Message);
                }
            }

            // listener.Stop();
        }
    }
}
and example client:
using System;
using System.IO;
using System.IO.Compression;
using System.Net.Sockets;

namespace LZ4.Test.Client
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            var host = "127.0.0.1";
            if (args.Length > 0) host = args[0];

            var client = new TcpClient();
            client.Connect(host, 4444);

            Console.WriteLine("Connected...");

            using (var tcpStream = client.GetStream())
            using (var bufStream = new BlockingBufferStream(tcpStream))
            using (var lz4Stream = new LZ4Stream(bufStream, CompressionMode.Decompress))
            using (var reader = new BinaryReader(lz4Stream))
            {
                while (true)
                {
                    var file = reader.ReadString();
                    if (file.Length == 0) break;
                    Console.WriteLine(file);
                    var length = reader.ReadInt32();
                    var bytes = reader.ReadBytes(length);
                }
            }

            Console.WriteLine("Done.");
            Console.ReadLine();
        }
    }
}
this additional element you need is BlockingBufferStream:
    public class BlockingBufferStream: Stream
    {
        private readonly Stream _innerStream;

        public BlockingBufferStream(Stream innerStream)
        {
            _innerStream = innerStream;
        }

        public override bool CanRead { get { return _innerStream.CanRead; } }
        public override bool CanSeek { get { return _innerStream.CanSeek; } }
        public override bool CanWrite { get { return _innerStream.CanWrite; } }
        public override bool CanTimeout { get { return _innerStream.CanTimeout; } }
        public override void Flush() { _innerStream.Flush(); }
        public override long Length { get { return _innerStream.Length; } }

        public override long Position
        {
            get { return _innerStream.Position; }
            set { _innerStream.Position = value; }
        }

        public override int Read(byte[] buffer, int offset, int count)
        {
            var total = 0;
            while (count > 0)
            {
                var chunk = _innerStream.Read(buffer, offset, count);
                if (chunk == 0 && total == 0) return 0;
                offset += chunk;
                count -= chunk;
                total += chunk;
            }
            return total;
        }

        public override long Seek(long offset, SeekOrigin origin)
        {
            return _innerStream.Seek(offset, origin);
        }

        public override void SetLength(long value)
        {
            _innerStream.SetLength(value);
        }

        public override void Write(byte[] buffer, int offset, int count)
        {
            _innerStream.Write(buffer, offset, count);
        }

        public override int ReadByte()
        {
            return _innerStream.ReadByte();
        }

        public override void WriteByte(byte value)
        {
            _innerStream.WriteByte(value);
        }

        public override void Close()
        {
            _innerStream.Close();
            base.Close();
        }

        protected override void Dispose(bool disposing)
        {
            if (disposing)
                _innerStream.Dispose();
            base.Dispose(disposing);
        }
    }
Aug 20, 2013 at 4:05 AM
Edited Aug 20, 2013 at 4:24 AM
Hmm doesn´t work, think i messed something up though, so probably my fault.

But i don´t understand one thing in your code. Why does the Client Decompress and the Server Compress?
Isn´t it supposed to be the other way around?

Thanks

EDIT:

Well it does work with normal Stream... But i get that it has "Forcibly Refused Connection" for some reason.
hmm.

Well it seems like before, if i change the blocksize to about 1024, everything works, but only for Jpeg.
if i change to .Bmp it "works" but is extremely slow.

Not sure why i have to change the blocksize:S
Aug 20, 2013 at 8:12 AM
This is just an example.
Server waits for clients. Clients connect, server compresses given folder and sends files over TCP to client. Clients decompresses them and does... well.. nothing in this examples.
This example is about just about compressed transmission over TCP. And it DOESN'T work with small block size. Well it maybe does but it's a fluke, not a fix. Although adding BlockingBufferedStream on a Client (well, READING side) does fix the problem.
Aug 20, 2013 at 8:31 AM
Okay, well i used it as you showed i think.

Meaning, i take my screenshot, then send it throught the Blockbuffstream, and then receive it with a normal lz4stream.

So it´s the same except i added the Blockbuffstream.

And well, it didn´t work.

Hmm, but then again, it may be better to just compress it while i am saving the image.

Though i have tried that and didn´t succeed, so will show you:

internal static void CaptureBitBit(IntPtr hwnd, EncoderParameters JpegParam, ImageCodecInfo jpeg, LZ4Stream ms)
    {
        Rect rc;
        GetWindowRect(hwnd, out rc);

        using (Bitmap bmp = new Bitmap(rc.Width, rc.Height, System.Drawing.Imaging.PixelFormat.Format32bppArgb))
        {
            using (Graphics g = Graphics.FromImage(bmp))
            {
                IntPtr dc1 = g.GetHdc(); 
                IntPtr dc2 = GetWindowDC(hwnd);
                try
                {
                    BitBlt(dc1, 0, 0, rc.Width, rc.Height, dc2, 0, 0, 13369376);
                }
                finally
                {
                    g.ReleaseHdc(dc1);
                }
            }

            //bmp.Save(ms, ImageFormat.Bmp);

            bmp.Save(ms, jpeg, JpegParam);
        }
    }
So i have a LZ4Steam, that will write to a memorystream, like this:

var ms = new MemoryStream();
LZ4Stream LZ4 = new LZ4Stream(ms,Compress,false);


So if i understand this right.
If i tell the bmp to save to the LZ4Stream, it will save it to the MemoryStream compressed.

Is this correct?
Aug 20, 2013 at 2:57 PM
But it seems to me like you don't need LZ4Stream.
You just need LZ4Codec.Encode(byte[]) and LZ4Codec.Decode(byte[]).
Lesve LZ4Stream alone.
Aug 20, 2013 at 3:22 PM
Depends, the thing is.
If i use Encode Decode byte (which i prefer, it´s easy), i have to double the work.
At least from my knowledge of how to do it, for example.

i have an image named bmp.

bmp.save(ms,blabla,blabla)

then that´s the image save into a memorystream.
But it´s not LZ4 compressed. So to do it with byte, i have to redo that process(or well not redo, but kinda).

new MemorysStream(LZ4Codec.Encode(ms.getbuffer()));


So i have to encode the memorystream buffer, and put that into a new memorystream.

So, why do that, if i can save it encoded into the memorystream at first?


Maybe i am doing it wrong, don´t know, but that´s at least how i understand it.
Sep 3, 2013 at 3:51 PM
Edited Sep 3, 2013 at 3:53 PM
Back from holiday.
Version 1.0.3.93 fixes this problem.
(you still require "using" on LZ4Stream, as you do "using (var tcp = new TcpClient())")