This project has moved. For the latest updates, please go here.

Remote.Linq - compress returned data

Apr 24, 2015 at 9:50 AM
Hello,
Would be possible to compress the data exchanged via WCF when using RemoteLinq ? Some queries generated from the application return a really huge number of records from the database.
We actually use the
IEnumerable<DynamicObject> ExecuteWithEntityFramework(...)
method on WCF server side.
Is it somehow possible to compress the generated DynamicObjects ?
Thanks in advance.
Coordinator
Apr 24, 2015 at 10:15 AM
Sure you can :-)

Remote linq does not even dictate the communication technology. The idea is that you can use whatever protocol and data formatting you prefer.

Using WCF, first of all you decide on the service contract to be used. Hence, you could even just use byte-arrays to submit query expressions and/or query results, holding compressed data. Second, and probably preferred, you can use compression with standard WCF binary encoder.
Apr 24, 2015 at 1:21 PM
I tried to replace the WCF DataContractSerializer with the protobuf-net serializer for the WCF endpoint. I am getting an exception when calling the ExecuteQuery() method, as probably the Remote.Linq.Expressions.Expression type cannot be serialized by protobuf.
Coordinator
Apr 25, 2015 at 11:29 PM
Edited May 7, 2015 at 9:41 PM
I’ve created a new branch (protobuf-net) and solution (Remote.Linq.protobuf-net) to add protobuf-net configuration for remote linq.

However, there is a limitation with protobuf-net which seems to be an issue which I further need to investigate…

Take the following proto contract as an example:
[ProtoContract]
public class Data
{
    [ProtoMember(1, DynamicType = true)]
    public object Value { get; set; }
}

[ProtoContract]
public class MyMessage
{
    [ProtoMember(1)]
    public string Text { get; set; }
}
Even though protobuf-net supports dynamic properties (DynamicType=true) it currently cannot handle primitive values in this case. That means the Value property in the sample above can be serialized when containing a reference type like MyMessage, but not when containing primitive values of type int, double, etc. See stackoverflow.

This is an issue since remote linq is using dynamic structures for query results and parts of query expressions.

The issues found are documented by failing unit tests in the Remote.Linq.protobuf-net solution.
Apr 27, 2015 at 5:56 PM
Edited Apr 27, 2015 at 5:57 PM
Thank you for the reply, but unfortunately I am afraid I cannot follow you.
We are actually using Remote.Linq to transport dynamic expressions via WCF, to our WCF service, which uses EntityFramework to access the database, execute the WCF client queries and return the dynamic objects back to the WCF clients.
As I wrote above, we basically call the ExecuteWithEntityFramework() method on a RemoteLinq Expression send via WCF, and returning an IEnumerable with Remote.Linq DynamicObjects back to the WCF caller.
Could you please post a code sample about how to add the protobuf-net support to the scenario above ? Or, if I correctly understand your posting, it is not actually possible to use it for such a purpose, as there are lots of primitive values used to represent the database columns in the different entity classes generated for the corresponding database tables ?
Coordinator
Apr 28, 2015 at 9:26 AM
Thanks for your feedback. I’ve added some more information to my explanation above. Please let me know whether things got clear.

I’ll post more info once I’ve done further research…
Apr 28, 2015 at 4:29 PM
Thanks for the explanation, now its clear that protobuf-net cannot work with Remote.Linq right now. I think that protobuf-net is mainly interesting for the query results (which can be huge amounts of data), and less for the expressions, which are rather small. As alternative, it would be interesting to know if the query results could be compressed with something like SharpZip.
Coordinator
May 28, 2015 at 2:53 PM
Sorry for the delayed response. There are mainly two options that would apply:
  1. Use some sort of compression mechanism supported by your communication protocol and communication framework of choice (e.g. WCF)
  2. Compress/decompress the query result using your own mechanism. You have total control, since you implement the service yourself. For your convenience I’ve created a sample application to demonstrate this idea.
Jun 10, 2015 at 3:19 PM
Thank you for the code sample, the second option looks like a good solution to reduce the amount of data containing the query result transported over network !