May 30, 2017

User-Guided Extraction

User-Guided Extraction (UGE) is a Downie feature that was introduced in Downie 2.2 and allowed Downie to download content it wouldn’t be able to download otherwise as the path to the content contained encryption, massive JavaScript execution and so on.

UGE is a way for you to download videos that aren’t directly supported by Downie out of the box. It is a small web browser that allows you to load the webpage in Downie while it keeps scanning the content that gets loaded by the webpage, listing content that it can handle.

You can find on how to use UGE by selecting Downie Help from the Help menu – there is an entire chapter on UGE. Which will cover most of the basic use of UGE, however, this article will focus on advanced use.

Several things to keep in mind

Playback

The fact that Downie is scanning for content that the web page loads implies one important thing that one needs to keep in mind when using UGE – you may need to start playing the video (or audio) in order to get the video you want. As Downie scans content that gets loaded by the webpage, the actual video may not get loaded until you press the play button.

Flash Player

As the web component that Downie uses is the same as in Safari, if the webpage requires Flash Player in order to play the content, you need to have it installed. This usually affects Chrome users as Chrome contains an embedded version of Flash Player that is not available to Downie.

iOS Mode

In the top-right corner, there is a switch between iOS and macOS modes. In iOS mode, Downie pretends to be Safari on iOS, disabling all plugins. If the web page has a mobile mode, it will most likely be used, including HTML5 playback which doesn’t require Flash.

Selecting the Correct Result

Now we are getting to the interesting part – getting the most out of UGE. On most sites, Downie will list one or two results in the right column, however, there are sites where the list will almost get flooded and it can be hard to select the correct result.

Firstly, let’s shed some light on why does Downie find so many results in some cases. There are three main reasons for this:

  • As mentioned in the previous blog post, Downie recognizes link patterns. Unfortunately, some sites have very non-specific pattern so that almost any link from that site gets matched. Each site nowadays consists from dozens of various files.
  • HLS/HDS streams consist of many small files. Downie tries to filter those by some heuristics, but that doesn’t always work.
  • There is a lot of content on that page – various videos, GIF animated images, etc.

When examining the videos, it’s important to notice that three lines for each result. The top line contains a name, which can be either name of the file, or title of the webpage, depending on various aspects.

The third line contains the server that hosts the file.

As you have noticed, I’ve just skipped the second line, which is most important. It contains the type of the download. It may say YouTube, Vimeo, etc.; or “Direct Downloader”. And this is very important, Direct Downloader is used for actual media files (MP4, MP3, …) – and that’s most likely what you want.

Starting Downie 2.9 (to be released a week from the time of writing this blog post), this line will contain size and type of the file as well. Pay special attention to this as you will want files that are size-wise related to the length of the video – if the video is 20 minutes long, a 3MB file is likely to be just a pre-roll video, or an ad, not the actual video content.

HLS streams (shown as “MP4 Stream” in UGE) do not contain size information as their size cannot be easily determined because they can consist of hundreds of segment files.

With the streams, you may get several of them. This is because the first one is likely the general manifest that describes which resolutions are available – each resolution then has its own manifest. You will get best results selecting the topmost link that says “MP4 Stream” as its format as Downie will pick the resolution for you based on your settings.

Summary

To quickly sum it up, unless there is an embedded video from a supported site (YouTube, Vimeo, …), you should be looking for “Direct Downloader” download. If there are several of them, select one based on the size and format, in case MP4 Streams, select the one at the very top as that is most likely the manifest root.

If you find yourself more adventurous, hover your mouse over the result to get a tooltip with the full link.

  • facebook
  • googleplus
  • twitter
  • linkedin
  • linkedin
Previous «
Next »