Categories
REST

Decompressing the TRESTResponse

RAD Studio, Delphi and C++ 10 Seattle has a fantastic REST Client Library. If you are unfamiliar with it, check out my 5 Minute REST Client video.

The TRESTRequest component has an AcceptEncoding property where you can specify the type of compression you would like applied to the response. The TRESTResponse component has a ContentEncoding property that specifies the the encoding that the server chose. The most common compression encoding is gzip, followed by deflate. The others are rarely used (although brotli is a new compression that looks promising for the future.)

By default it is not encrypted if you leave AcceptEncoding blank. However when you use the Stack Overflow API it always compresses with gzip if nothing is specified.

The TRESTResponse has two ways to access the response data directly, the RawBytes property and the Content property. Both are read-only. So we can read the compressed response data, but cannot update the TRESTResponse with the decompressed data. Updating the TRESTReponse allows it to work with the TRESTResponseDataSetAdapter, which is a fantastic feature.

Luckily we can use the protected mode workaround to update the value. Here is a unit that you can include in your project to easily decompress the TRESTResponse and continue to use it with the Data Set Adapter.

unit RestDecompress;

interface

uses
  System.SysUtils, System.Types, System.Classes, IPPeerClient, REST.Client;

procedure DecodeRestResponse(ARestResponse: TRESTResponse);

implementation

uses
  System.Zlib,
  IdBaseComponent, IdException, IdZLibCompressorBase, IdCompressorZLib;

  type // protected mode work around
  TProtectedRESTResponse = class(TRESTResponse)
  end;


procedure DecodeRestResponse(ARestResponse: TRESTResponse);
var
  LCompressed: TMemoryStream;
  LDecompressed: TStringStream;
  LDecompress: TIdCompressorZLib;
begin
  if Length(ARestResponse.ContentEncoding) = 0 then exit;

  LCompressed := nil;
  LDecompressed := nil;
  LDecompress := nil;
  try
    LCompressed := TMemoryStream.Create;
    LDecompressed := TStringStream.Create;

    LCompressed.WriteData(ARESTResponse.RawBytes, Length(ARESTResponse.RawBytes));
    LCompressed.Position := 0;

    // Use the Indy decompression libraries because the HTTP stream doesn't
    //   have the proper headers that System.ZLib looks for.

    LDecompress :=TIdCompressorZLib.Create();

    if ARestResponse.ContentEncoding = 'gzip' then
      LDecompress.DecompressGZipStream(LCompressed, LDecompressed)
    else if ARestResponse.ContentEncoding = 'deflate' then
    begin
      // Due to variations in the deflate server side implementations,
      //   this rarely works, but is here for completeness and just in case
      LDecompress.DecompressHTTPDeflate(LCompressed, LDecompressed);
    end;

    TProtectedRESTResponse(ARESTResponse).SetContent(LDecompressed.DataString);
  finally
    LDecompressed.Free;
    LCompressed.Free;
    LDecompress.Free;
  end;
end;

end.

Simply use this unit, and then add a call to DecodeRestResponse(RESTResponse) to RESTResponse’s OnAfterExecute event handler. It checks the ContentEncoding and then uses the correct decompression and updates the content.

You’ll notice it uses the Indy TIdCompressorZLib component instead of the new System.ZLib library. The reason is a GZip encoded HTTP response doesn’t include the full headers expected by the ZLib library. There is a way to work around this, but no need to do that since the Indy library works fine.

Categories
REST Tools

Delphi and REST Client Development

I really don’t like to make comparison between different development tools. In my opinion most tools have their strengths for certain tasks. So I try not to criticize other tools by name. Delphi is my favorite, and I believe it is the best all around tool for more projects I face. Occasionally I have someone ask me how Delphi compares to another tool they are evaluating. I thought I would share one such experience.

When I was in Brazil I went to meet with a guy who was in charge of software development for a large, international company. They were all standardized on a different development tool, but it didn’t really have a mobile solution. They were looking at some tools that claimed to make it really easy to do mobile development, but once he got past the marketing he found it wasn’t all it was cracked up to be.

To evaluate the tool he decided he would develop a simple little app for iPhone and Android that connected to a REST server to pull down a list of countries, cities, etc. to help people find offices for his company. He thought that a week should be more then enough time to develop this simple application.

What he found was that while he expected it to be cross platform, it was actually only shared code. So he had to create a project for each platform, and then he could share part of the code (not as much as he would like) between each project. Also there was significant overhead in consuming a REST service. After a couple days he realized a week wasn’t enough time for even one platform.

To top it all off he discovered that the cost was actually way more then it was initially made out to be when you actually got down to doing something serious.

So I showed him Delphi and told him that the app he was describing could be done inside a day, probably with a prototype within an hour. He found that a little hard to believe, so I created a simple REST client displaying the result of a REST service and deployed it to my Android phone. It took about 5 minutes, including the time it took to connect to his WiFi.

He was so impressed he actually got his phone out to make a video of it just so he could show his boss, who he was sure wouldn’t believe him.

That is what I love about Delphi. It makes the simple things that you need to do all the time so simple and so quick, and it makes everything else still pretty simple. Need an API that isn’t wrapped in a component or the RTL, no problem, you can still access it.

Categories
DataBase REST Tools

Export Trello to CSV

I’m a big fan of Trello as a way to organize and collaborate. Recently I needed to export a Trello board to CSV. They have an export to JSON, but you have to buy a years worth of business class for the export to CSV. The business class is probably worth it for other reasons as well, but I thought I would see what it would take to convert JSON to CSV using Delphi 10 Seattle’s REST Client and FireDAC. I’m really pleased with what I came up with, and it didn’t take much effort.

I used the REST Client to connect to the JSON export on Trello. The REST Client is overkill for this since it is just a simple HTTP GET request, but the REST Response DataSet adapter is what I was after. This piped the JSON into a TFDMemTable.

Now the JSON from Trello contains a few collections (JSON Array’s of objects) so I used 3 different adapters (each with a different root node) and 3 different memory tables: Cards, Lists and Labels. There are other collections, but I wasn’t interested in them right now.

Once I have the JSON in a DataSet, I need to work with it. So I put down a TFDLocalSQL component and pointed it to the Mem Tables. Then attached a TMemo to the Query of a TFDQuery component, and I was able to run SQL querries against the Trello board. This made it easy to select exactly the data I wanted to export.

For the export I just used the FireDAC ETL functionality to pump the DataSet resulting from the Local SQL query to a text writer which saves a CSV file.  I love the way FireDAC provides all this great high level functionality like Local SQL and the Text Writers. It makes so many tasks so easy to deal with.

There is still a lot of polish to do for this, but right now it does the job really well. I thought I would share it with anyone else who might be interested in exporting Trello to CSV. You could take a look at the source code and use it to convert most any REST/JSON datasource to CSV.

I put the code up on GitHub.

Categories
REST Tools

REST and the Kimono

I love the new TRESTClient components in XE5. Especially the ability to visual bind a REST datasource with the use of the TRESTResponseDataSetAdapter. Now I find I’m always on the look out for new REST datasources. The REST Debugger makes the whole process really easy too.

The other day I hit the mother-load of REST datasources with Kimono Labs. It is a creative web service that makes it easy to scrape a web site and turn it into a REST data source. It looks for repeating data on the page. Their free service is enough to get you started. I created a simple REST datasource of San Francisco 49ers games from their schedule on their web site.

kimonolabs.comA few tips for working with the Kimonolabs REST API.

  • Make sure the web data doesn’t include hyperlinks – if it does, then the REST data will include objects containing the href and the text, which doesn’t map to a grid well.
  • Use results.collection1 as your root element in the TRESTResponseDataSetAdapter and you are off to the races.
  • It doesn’t work with every web page because of malformed pages, but it works with a lot of them.
  • There is no way to edit your API once you’ve finished it (yet), so you end up deleting and recreating it a few times.

They have a lot of videos and tutorials on their site walking you through how to use their service. Take a look and I’m looking forward to your REST enabled apps!

 

 

Categories
Delphi Projects REST Source Code

Delphi XE5 Mobile REST Client Demo Source

With the release of Delphi XE5 I’ve made the source of the Mobile REST Client Demo available. This is a really super simple demo of the Mobile REST Client in XE5. It is designed to show up how you can consume a JSON REST Service and adapt it into a DataSet and then bind that to the UI. If you change the data source you will probably need to change the live binding.

I’ve changed the demo from the one I used in the video to consume an OData data source. OData is a new standard backed by Microsoft for sharing data over the web. You can think of it as SQL for the web. It is a combination of other technologies, including REST, AtomPub, and JSON. Sybase supports OData on all of their databases now, as does Microsoft and others.

The change was simply a matter of pointing it to the Northwind OData endpoint provided on Odata.org. The binding is setup to show the company name in the listview. I added the ability to specify a Root Element, but that isn’t necessary for the demo.

This new technology works in both desktop and mobile, FireMonkey and VCL. It should also work in C++ Builder.

[Download the demo] (MobileRestXE5.7z 8 KB – Requires Delphi XE5 – Builds for iOS, Android or Win32)

Categories
Audio podCast podcast

38 – Marco & Cary

Visiting with Marco Cantu and Cary Jensen.

If you have any other questions or comments for Marco or Cary you can leave them here, or catch them online:

Marco Cantu

Cary Jensen

Categories
News

Marco Cantu’s REST Webinar

I didn’t get the episode with Marco and Cary edited and posted yet (came close – might still get it posted tonight when I get home).  In that episode Marco mentioned his free REST webinar which occurs at three different times tomorrow (March 3rd).  So sign-up to attend one of those sessions and download the white paper, it will be really good.
Marco Cantù presents REST in Delphi and RAD Studio 2010

Let Marco Cantù show you how to get the most from RAD Studio 2010 with Representational State Transfer (REST) – a new architecture for Web services that is having a significant impact on the industry. Join Marco as he delves into the technologies involved in REST from the Delphi perspective.

Register now for
March 3 6:00 AM PST / 3:00 PM CET

March 3 11:00 AM PST / 2:00 PM EST

March 3 8:00 PM PST / March 4 3:00 PM AEST

Topics covered in this webinar will include:

  • The Concepts Behind Representational State Transfer
  • REST technologies and Delphi
  • Creating Delphi REST clients that interface with RSS feeds, maps, and Twitter
  • Data-Oriented REST Servers