Byte Array serialization

Hi rpgmaker,
first of all thanks for your great work!

I (ab)-used a lot of your work for an inter-server communication project and I appreciated it.
Now I'm facing again my caching project, which I pulled out 2 years ago.
Updated libraries via NUGet and... go for benchmarking.
Well, NetJSON won by a mile over Wire. :) At least on my simple DTOs. :)

The problem is when I pass an object with byte array as property. I don't catch an error until I deserialize the resulting bytes.
To do so I encapsulated the Serialize and Deserialize in those two methods (they both inherit from an interface I use on my project, which I wish to don't change):

    public byte[] Serialize<T>(T value)
    {
        return Encoding.UTF8.GetBytes(NetJSON.NetJSON.Serialize(value.GetType(), value));
    }


    public T Deserialize<T>(byte[] bytes)
    {
        return (T)NetJSON.NetJSON.Deserialize(typeof(T), Encoding.UTF8.GetString(bytes));
    }

Are they correct? Would you suggest another way to serialize/deserialize an object?

Are the byte arrays excluded from a re-work in NetJSON?
Do I need to serialize a byte array through NetJSON.NetJSON.RegisterTypeSerializer (Func <T, string> serializeFunc); ? And if yes how would you suggest it?

Thanks again 👍

Hi,

Thanks for the comments. In regards to your issue, could you post the class, and the JSON output? And are you getting an error such as invalid JSON exception by any chance. If so, I recently(today) fixed a bug for it which has not made it to nuget yet but is available in the branch.

Thanks,

This is the class, provided by helper classes;

[Serializable]
[System.Diagnostics.DebuggerStepThrough]
public class ComplexObject
{
    static Helper.RandomBufferGenerator generator = new Helper.RandomBufferGenerator(65000);

    public ComplexObject()
    {
        Thing1 = true;
        Thing2 = int.MaxValue;
        Thing3 = 'q';
        Thing4 = "asdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasas";
        Thing5 = new Dictionary<string, string>()
        {
            { "1", Helper.RandomString(4) },
            { "2", Helper.RandomString(4) },
        };
        Thing6 = generator.GenerateBufferFromSeed(32000);
        Thing7 = uint.MaxValue;
    }

    public bool Thing1 { get; set; }

    public int Thing2 { get; set; }

    public char Thing3 { get; set; }

    public string Thing4 { get; set; }

    public Dictionary<string, string> Thing5 { get; set; }

    public byte[] Thing6 { get; set; }

    public uint Thing7 { get; set; }
}

    [System.Diagnostics.DebuggerStepThrough]
    public class RandomBufferGenerator
    {
        private readonly Random _random = new Random();
        private readonly byte[] _seedBuffer;

        public RandomBufferGenerator(int maxBufferSize)
        {
            _seedBuffer = new byte[maxBufferSize];

            _random.NextBytes(_seedBuffer);
        }

        public byte[] GenerateBufferFromSeed(int size)
        {
            int randomWindow = _random.Next(0, size);

            byte[] buffer = new byte[size];

            Buffer.BlockCopy(_seedBuffer, randomWindow, buffer, 0, size - randomWindow);
            Buffer.BlockCopy(_seedBuffer, 0, buffer, size - randomWindow, randomWindow);

            return buffer;
        }
    }

    [System.Diagnostics.DebuggerStepThrough]
    public static string RandomString(int length)
    {
        string chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
        char[] stringChars = new char[length];
        Random random = new Random();
        
        for (int i = 0; i < stringChars.Length; i++)
        {
            stringChars[i] = chars[random.Next(chars.Length)];
        }

        return new String(stringChars);
    }

I don't get any exception. Serialization is fine, Deserialization too.
I pass the deserialized object to ShouldDeepEqual method provided by NUGet package DeepEqual v.1.6.0 ( https://github.com/jamesfoster/DeepEqual ) and I get an error.

I made a screenshot of the properties of the object (source and destination) to help on this.

Please note how property Thing3 is also flawed.

I'll try with the nightly build and let you know, thanks! :)

The new build, downloaded some minutes ago, didn't work. I see the same results.

Thanks

Thanks, I will look into it.

Hi, I will try to resolve it this weekend.

Thanks,

Thanks to you!

Thanks again for the find. The issue was not byte array, it was because my deserialization was not properly handling the logic that process "char", therefore it was destroying the result of char and at the same time your byte array was not matching because your constructor was setting it to a default value and then the deserialization never happened because the reading of the property name made the serializer to skip the property.

I have published it to nuget. Please test it again

Thanks,

The fix didn't solve the problems.... the fix seems fine for a particular problem, except for the fact that I'm receveing a base64 encoded string at place of byte array.
I didn't convert it but I'm sure it's my byte array. :)

In the attachment the error I'm getting.

error.txt

I want to be sure I'm using NetJSON properly.
So I ask you again, to be sure... do you think I'm implemeting Serialize/Deserialize functions like you intended? They're on the first post of this thread.

I can't attach all my code, neither my unit test cause is explicitly testing against a whole project but I can attach the code in which I create the object graphs to test.

using DeepEqual.Syntax;
using Mercury.Cache.Serialization;
using Mercury.Cache.UnitTest.Data;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using NUnit.Framework;
using System.Collections;

namespace Mercury.Cache.UnitTest
{

    [TestFixture]
    public class serialization
    {
        readonly int NumberOfTests = 1000;

        static List<PerfStats> GlobalStats = new List<PerfStats>();

        class PerfStats
        {
            public Formatter Formatter { get; set; }
            public Type ObjectGraph { get; set; }
            public float ElapsedTime { get; set; }
            public long BufferSize { get; set; }
        }

        static Formatter[] Formatters = new Formatter[]
        {
            Formatter.BinaryFormatter, Formatter.CompactFormatter, Formatter.WireFormatter, Formatter.NetJsonFormatter
        };

        static object[] ObjectGraphs = new object[] {
            new ComplexObject(),
            new ComplexType(),
            new Dictionary<string, ComplexType>()
            {
                {"Key3", new ComplexType() { } }
            },
            new Helper.RandomBufferGenerator(65000).GenerateBufferFromSeed(32000)
        };


        // ------------------------------------------------------------------- //

        [OneTimeSetUp]
        public void WarmUp()
        {
        }


        [TearDown]
        public void CleanUp()
        {
            var stats1 = GlobalStats.GroupBy(x => x.ObjectGraph).ToList();
            Console.WriteLine("---[ Best performers for Elapsed Time ]---");
            foreach (var group in stats1)
            {
                Console.WriteLine("---[ " + group.Key + " ]---");
                foreach (var item in group.OrderBy(x => x.ElapsedTime))
                {
                    Console.WriteLine("Formatter: " + item.Formatter);
                    Console.WriteLine("ObjectGraph: " + item.ObjectGraph);
                    Console.WriteLine("Elapsed time: " + item.ElapsedTime + " ms");
                    Console.WriteLine("Size: " + item.BufferSize);
                    Console.WriteLine("------------------------------------------------------------------");
                }
                Console.WriteLine();
            }

        }


        [Test, Combinatorial]
        public void Test([ValueSource("Formatters")] Formatter formatter, [ValueSource("ObjectGraphs")] object objectToCache)
        {
            PerfStats result = new PerfStats();
            long sizeOfSerializedObject = 0;

            MultiSampleCodeTimer sample = new MultiSampleCodeTimer(NumberOfTests);
            sample.OnMeasureCompleted = (graph, interation, MSCD) =>
            {
                string[] names = graph as string[];
                result = new PerfStats()
                {
                    Formatter = (Formatter)Enum.Parse(typeof(Formatter), names[0]),
                    ObjectGraph = Type.GetType(names[1], false),
                    ElapsedTime = (float)Math.Round(MSCD.Mean, 4),
                    BufferSize = sizeOfSerializedObject
                };
            };

            sample.Measure(new string[] { formatter.ToString(), objectToCache.ToString() }, () =>
            {
                byte[] serialized = Engine.Serialize(objectToCache, formatter);

                object cachedObject = Engine.Deserialize<object>(serialized, formatter);

                if (sample.First)
                {
                    cachedObject.ShouldDeepEqual(objectToCache);
                    sizeOfSerializedObject = serialized.Length;
                }
            });


            GlobalStats.Add(result);
        }
    }
}

Please, let me know what portion of code do you need to test it yourself.

I Will take a look at it today. I tested using the original code that you posted and it works fine. Are you using the latest code?

I downloaded version 1.2.9 from NUGet. Is it right?

That is the latest. I have a unit test running your code and using deepequal working fine.

Are you specifically using byte array or object for serialization? Object will not deserialize complex object. Byte array been converted to string to deserialization should be fine as long as you are preserving the encoding used.

Can you post the specific logic that does the serialization and deserialization?

Thanks,

This is the wrapper around your NetJSON.

I'll do some code-digging to find out where the things are gone wrong.
Thanks for you precious help. :)

class NETJsonFormatter : IBinarySerializer
{
    public NETJsonFormatter() { }

    static bool Initialize()
    {
        NetJSON.NetJSON.IncludeFields = true;
        NetJSON.NetJSON.IncludeTypeInformation = true;
        NetJSON.NetJSON.UseEnumString = true;
        return true;
    }

    static bool Initialized = Initialize();


    public byte[] Serialize<T>(T value)
    {
        return Encoding.UTF8.GetBytes(NetJSON.NetJSON.Serialize(value.GetType(), value));
    }


    public T Deserialize<T>(byte[] bytes)
    {
        return (T)NetJSON.NetJSON.Deserialize(typeof(T), Encoding.UTF8.GetString(bytes));
    }


    public object Deserialize(Stream serializationStream)
    {
        return NetJSON.NetJSON.Deserialize(typeof(object), Encoding.UTF8.GetString(serializationStream.ToByteArray()));
    }


    public void Serialize(Stream serializationStream, object graph)
    {
        byte[] serialized = Encoding.UTF8.GetBytes(NetJSON.NetJSON.Serialize(graph.GetType(), graph));
        serializationStream.Write(serialized, 0, serialized.Length);
    }
}

You were right.

"Are you specifically using byte array or object for serialization? Object will not deserialize complex object. Byte array been converted to string to deserialization should be fine as long as you are preserving the encoding used."

Solved.
Thanks again!

Which part was right? 😀

:p

Please refer to the last Unit Test I've posted, which is the original unit test I've adapted to run multiple serializer at once.
I've made the assumption would be possible to do something like this:

            object objectToCache = new ComplexType();
            byte[] serialized = Engine.Serialize(objectToCache, formatter);
            object cachedObject = Engine.Deserialize<object>(serialized, formatter);
            cachedObject.ShouldDeepEqual(objectToCache);

which is wrong.
It's working fine for BinaryFormatter and Wire too, but not for NetJSON.
Anyway this is not a real case. I'll never ever in my code using something like this, I use generic type to cast against so the live code will result in something like this:

            ComplexType objectToCache = new ComplexType();
            byte[] serialized = Engine.Serialize(objectToCache, formatter);
            ComplexType cachedObject = Engine.Deserialize<ComplexType>(serialized, formatter);
            cachedObject.ShouldDeepEqual(objectToCache);

which will works.

I'm pretty unsure yet if my wrapper (class NETJsonFormatter, posted above) is wrong or do you think is fine.

The wrapper class looks fine to me. I would suggest using only "include type information" as a static property since I have not ported it to the netjson settings class. But you should create an instance of netjsonsetting for serialization and deserialization.

Thanks,

Ah ok, thanks.
Corrected the implementation like you suggested.

Thanks again! 🥇

Navigation Menu

Repository navigation

Byte Array serialization #150

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Byte Array serialization #150

Description

Activity

rpgmaker commented on Jan 23, 2017

LorenzoRuggeri commented on Jan 23, 2017

LorenzoRuggeri commented on Jan 23, 2017

rpgmaker commented on Jan 23, 2017

rpgmaker commented on Jan 25, 2017

LorenzoRuggeri commented on Jan 26, 2017

rpgmaker commented on Jan 30, 2017

LorenzoRuggeri commented on Jan 30, 2017

rpgmaker commented on Jan 30, 2017

LorenzoRuggeri commented on Jan 30, 2017

rpgmaker commented on Jan 30, 2017

rpgmaker commented on Jan 30, 2017

LorenzoRuggeri commented on Jan 30, 2017

LorenzoRuggeri commented on Jan 30, 2017

rpgmaker commented on Jan 30, 2017

LorenzoRuggeri commented on Jan 30, 2017

rpgmaker commented on Jan 30, 2017

LorenzoRuggeri commented on Jan 30, 2017

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions