How to YouTube

[heyrick]

Hi,

All this talk about deep linking to YouTube files. I've done some digging and it would appear that getting a video out of YouTube, given a video URL, is fairly simple really. Perhaps prone to minor changes if YouTube tweak stuff, but we ought to be able to sniff out how it works afterwards?


Anybody fancy poking the YouTube application or writing a YouTube "player"? I suggest writing a new one to work around the search API not being 'known' at this time, though perhaps a perusal of the published API might help?


Anyway, here are the results of my investigations.


We shall work with a rather sweet little Christmas song that is very popular in Japan. It dates from the mid-'80s. This is mostly an audio track, but the static (video) picture does change from time to time.
  http://www.youtube.com/watch?v=k8QsZCHhZeY
  [ TATSURO YAMA[chocolate ice cream]A - Christmas Eve(1983) ]

^^ note the "chocolate ice cream". His name is Ya-ma-shi-ta, the obscenity parser is just being a bit enthusiastic. I wonder what it would do with Scun...thorpe (a real British town).


Okay. The first thing we need to do is enquire this video. For this, we call:
http://www.youtube.com/get_video_info?&video_id=k8QsZCHhZeY&el=embedded&ps=default
where the ID is extracted and placed after &video_id=.

This will return an epic amount of gobbledegook, most of it is no use whatsoever to us.

Don't bother parsing the available formats, we will be forcing format 5 (old style FLV with 24 lines max and MP3 audio (frequently mono, thankfully not in this case!).

What is important to us is the "token", perhaps to make it a little more complicated for auto-rippers.
This is in the wodge of data thrown to us by the get_video_info call. Just search for "&token=" and pull everything until the next '&'. The token for this video is:
  vjVQa1PpcFNQoZDma1YJlw0QOdH7AMR0yfcuRp0SRdk%3D
I did this using Firefox. On the OSD, we probably need to parse "%3D" to '='.
This is then assembled into a call to retrieve the video:

http://www.youtube.com/get_video?video_id=<id>&t=<token>&eurl=&el=&ps=&asv=&fmt=5

Where the <id> is the video ID as seen in YouTube URLs, and <token> is the string of gibberish we just parsed out of the info file.

&fmt=5 forces us to 240p mode. It is my experience that pretty much every video has a 240p (inc. those that don't claim to).
  Re: http://en.wikipedia.org/wiki/YouTube#Quality_and_codecs


The complete URL is, thus:
http://www.youtube.com/get_video?video_id=k8QsZCHhZeY&t=vjVQa1PpcFNQoZDma1YJlw0QOdH7AMR0yfcuRp0SRdk%3D&eurl=&el=&ps=&asv=&fmt=5

This will return a file called "videoplayback" (no extension).


This may not work, if you click the link above. I don't know if the token is fixed per video, or based upon the current time, or your IP address, or what.


I saved this to my harddisc, gave it an FLV extension, and it plays fine in MPlayer. The specifics of this file are as follows:
--8<--------
General
Complete name                    : \\Azumi\Misc\Downloads\videoplayback.flv
Format                           : Flash Video
File size                        : 4.50 MiB
Duration                         : 4mn 15s
Overall bit rate                 : 148 Kbps
httphostheader                   : v2.cache3.c.youtube.com

Video
Format                           : H.263
Duration                         : 4mn 15s
Bit rate                         : 80.4 Kbps
Width                            : 320 pixels
Height                           : 240 pixels
Display aspect ratio             : 4:3
Frame rate mode                  : Constant
Frame rate                       : 29.970 fps
Bits/(Pixel*Frame)               : 0.035
Stream size                      : 2.45 MiB (55%)

Audio
Format                           : MPEG Audio
Format version                   : Version 2
Format profile                   : Layer 3
Mode                             : Joint stereo
Mode extension                   : MS Stereo
Duration                         : 4mn 15s
Bit rate mode                    : Variable
Bit rate                         : 58.9 Kbps
Channel(s)                       : 2 channels
Sampling rate                    : 22.05 KHz
Bit depth                        : 16 bits
Stream size                      : 1.80 MiB (40%)
--8<--------


Just thrown it across the LAN to the OSD. It is playing right now (as I write this).


I am starting to get irritated that I don't have a build environment running (and that I have no clue about Qt libs), for it looks like a real basic (user-provided URL) YouTube player could be built from a couple of fetches, a braindead parser, and then throwing the lot to the video playback library.
I don't know if you need to have the entire file in order to begin playback. To be honest, I would be inclined to download to the default location (USB or SD as applicable) and play back once the file is there. This will make the OSD a lot more resiliant to momentary hiccups in the download that would affect realtime playback.


Now, for verification, let's try this again. A real video this time.


The video chosen this time is:
  http://www.youtube.com/watch?v=GcJBFQ_2wbQ
  [ ZONE - Secret Base (Kimi ga kureta mono) english subtitled.avi ]

Read the subs, it's quite a sad song even though it doesn't sound it.

The info here is:
http://www.youtube.com/get_video_info?&video_id=GcJBFQ_2wbQ&el=embedded&ps=default

A bit of wetware parsing by eyeballing the file in Metapad, token is:
  vjVQa1PpcFNxE7455ETMDbjzBrxXVppI2hsuMHb3t4w%3D

Therefore:
http://www.youtube.com/get_video?video_id=GcJBFQ_2wbQ&t=vjVQa1PpcFNxE7455ETMDbjzBrxXVppI2hsuMHb3t4w%3D&eurl=&el=&ps=&asv=&fmt=5

It was supposed to take about a minute, it took nearly four as the speed kept bouncing from 200K/sec (not bad, 260 is maxing out for me) to a less impressive 30K/sec. If somebody writes this proggy, please add a K/sec calculation, it is something to look at. 

"videoplayback" -> "zone_secretbase.flv"

Tossed to the OSD.

Playing now.


There's an even cuter (MAKO - yes!) version of this song (audio only, static picture) at:
  http://www.youtube.com/watch?v=LS0s1sUxztY


Well. It's 4.40am here. I'm off to grab a Kitkat, watch the the most recent episode of "Ore No Imouto Ga Konna Ni Kawaii Wake Ga Nai" and then go to bed.

Hope this info is useful to somebody.


Best wishes,

Rick.

[greyback]

Hey Rick,
this I already know, and I wrote code to implement it some time ago.

Sadly it always crashed, for some reason I never could figure out, and then I got distracted by other things. If I can find the code, I'll upload it.
-G

[heyrick]

this I already know,

Meh. :'(

It isn't really documented anywhere. It's kinda funny looking at the YouTube/Google forums. As soon as somebody mentions the magic words ( /get_video?... ), it becomes "This topic is now closed.". Right, Google, so much for the hacker spirit. All those whizzy YouTube applets are great... if you have a whizzy thing to use them on. The OSD doesn't.


BTW, I've later done some testing and it looks like the token is related to both the time and the IP address.

and I wrote code to implement it some time ago.

As I would like to do sometime.

Sadly it always crashed, for some reason I never could figure out,

Symptoms? Fetch? Playback?

and then I got distracted by other things.

<cough> Yeah...... I've rather a lot of half-finished projects here myself. I guess the awesome part about coding is to create something. But debugging is this horrible stage where you're doing more work for less obvious results.

If I can find the code, I'll upload it.

Thanks. And please include an executable (even if crashy). It'd be interesting to know at what point it does go wrong, narrow down what's going on.


Best wishes,

Rick.

[greyback]

A zip file of what I have done is available here. Note it was a year ago now, so it may not work even now!

In the "lib" folder is my version of the library Neuros wrote to parse stuff from YouTube. - the old broken version sits in ...../linux-r3-main-app/youtube/src/lib - I planned to correct it and just insert my fixed version - which I did, but crashy crashy!

The zip file also contains two testcase apps. Testcases are super-simple, they doesn't do any remote access, just wants to open an XML file that Youtube sends you on API request (which I captured & saved to disk - some are in that zipfile too) and obtain the info it wants.

At the time, I was able to search Youtube and list stuff. But trying to play a video caused the playback process to be crash, or be killed.

I never found the time to set up a debugger on the OSD, so I'm not sure why it crashes. My theory was that my code isn't too soft on memory. I require the whole youtube HTML page source be downloaded into memory before parsing it for the video URL. I guessed the OSD killed the process for using too much memory. Maybe I've a leak. I dunno.

Welcome to try it
-G

[heyrick]

Hi.

Super-quick. The Internet plugs into the Livebox (a DSL router), which is in the living room. This uses smoke'n'magic (aka WiFi) to send the data to my computer.

The LAN, the Intranet, is in my bedroom. The OSD is there as well, connected to the LAN. I can plug into the LAN by popping the RJ45 into my computer. Then I have access to both networks.

Now... Um... Getting the OSD to plug into the Livebox is a little tricky. It was okay for greyback's testing because I just plugged it in and left it alone. But actually using it... that's a different kettle of fish! Computer, capture card, untangling the A/V cables from the SCART switcher. <panic!>

Is there a simple icky quick'n'dirty way to get the OSD to talk to the internet via my computer? Some sort of "proxy" perhaps? It can then LAN to me, and I'd bounce the requests onto the live network. Dunno if YouTube (etc) would work like that, but it'd be a start.


Best wishes,

Rick.

[greyback]

Yep it's possible. You want your PC to be a "wireless bridge" for the OSD. Google that. You might need to set up manual IP addresses for your network card & OSD tho.
-G

[heyrick]

Yep it's possible. You want your PC to be a "wireless bridge" for the OSD. Google that. You might need to set up manual IP addresses for your network card & OSD tho.

Okay.

It's like this.
Intranet. Heart is a DLink DIR-100 providing DHCP to all connected devices. That would be:

• Router - 192.168.0.99
• my eeePC [Azumi] (192.168.0.14)
• the OSD (192.168.0.12)
• Ayleigh (192.168.0. [1.1GHz AMD with big harddisc, for NAS-like shares]
• Aiko (192.168.0.9) [450MHz P2, old main computer]
• Alyson (192.168.0.1) [Acorn RiscPC 700]

As the router only has four ports, the last two are normally disconnected and one/other is plugged in as required.

Internet. Heart is either an Orange Livebox Mini (1.2) or a LinkSys WAG200G (running OpenWAG) depending on how the phone line is behaving. By preference the Livebox, as it has the landline phone (via VoIP).
Only one machine has regular access, and that is my eeePC (192.168.1.10). The router is 192.168.1.1 (Livebox) or 192.168.1.99 (WAG200).

In all cases, the eeePC and OSD use pseudo DHCP. This means it is real DHCP, but with the router(s) set up to 'reserve' specific addresses so the allocated IP is always the same.

Using XP to "bridge" the connections was pretty simple. Getting it to actually work was not such a walk in the park. The first thing, it doesn't seem able to bridge 192.168.0.x to 192.168.1.x so I reconfigured the LAN to be x.x.1.x.
The next thing was to faff around to set the gateway to 192.168.1.1 (Livebox) instead of the router.
At this point, the OSD was able to look up DNS records. Trying to ping Google got the DNS, but the ping failed. Hell, trying to ping the eeePC failed - even with the OSD in the DMZ.
I noted some oddities like Windows flagging the ethernet cable as being unplugged.
Yup. That'd be the same one I was using at that moment to telnet into the OSD. Like... huh?

Sadly the DIR-100 didn't "recognise" the PC when it was plugged into the WAN port. I dunno, perhaps it is trying to be too clever. Ping packets just wouldn't get out... so I gave up, reset everything back to normal...


The OSD has a WiFi function (or is this specific to OSDng?). I assume it is a USB dongle, yes? Is it a specific USB dongle? Or any generic one? I can rummage around eBay to see if I can pick up such a thing. Might be simplest.
It would need to support WPA2/AES, by the way.


Until then, I guess I'll need to just plug the thing into the Livebox as I did for your tests.


Best wishes,

Rick.

[edited to tiny up some screwy markup in the list (too many list opening tags, oops!)]

[ChadV]

The OSD WiFi is one specific range of wireless bridges.

If the OSD is next to a computer that connects via wireless anyway, you're better off using that bridged connection, though it sounds like you've a bit of a run around to get it to work so far.

Also, your "pseudo DCHP" is called Static DHCP. 

[heyrick]

Sadly, as of last month, this no longer works. Bloody YouTube!

The initial part, reading the video info and pulling the token, works just fine.
Unfortunately, the next part - retrieving the video - just returns a blank page when performed in the browser, or a 404 when done directly via telnet.

For the moment, I don't have the time to WireShark my connection to see what is actually going on...


Best wishes,

Rick.

[heyrick]

I was going to post that I've not had time to look at this... but I've cracked it. Wasn't that hard, just damned fiddly.

Okay.

The video is me walking my cat along the driveway (figured you guys were sick of J-Pop!).

This is just rough'n'ready, okay?

Step one - get the video info file.

http://www.youtube.com/get_video_info?&video_id=ba7uZ1koImU&el=embedded&ps=default

This isn't hard, just pop the YouTube video ID into the &video_id parameter.


It will reply a scary shedload of stuff.

Notice it says "url_encoded_fmt_stream_map" at the top. This means stuff is encoded several times over and will need cut'n'paste fixing.

First, search for "itag%253D5". The file appears to provide separate URLs and access keys for different qualities, perhaps in an attempt to handle load balancing? The type 5 (naffest quality around) is all that the OSD can cope with, so that's what we'll stick to.

Once you have found it, backtrack until you can see "url%3Dhttp". Keep the http part, everything before and including can be discarded.

Now look for "%26fallback_host". That, and everything following, can be discarded.

You should be left with a file that looks like this:
   http%253A%252F%252Fo-o.preferred.orange-par1.v2.lscache1.c.youtube.co
   m%252Fvideoplayback%253Fupn%253DnRoni3Eqzu4%2526sparams%253Dalgorithm
   %25252Cburst%25252Ccp%25252Cfactor%25252Cid%25252Cip%25252Cipbits%252
   52Citag%25252Csource%25252Cupn%25252Cexpire%2526fexp%253D909903%25252
   C901604%2526algorithm%253Dthrottle-factor%2526itag%253D5%2526ip%253D9
   0.0.0.0%2526burst%253D40%2526sver%253D3%2526signature%253D13C8AE4B622
   8BD8B7E4BD68EBD71DCEC7B5951D0.C326D73C133C465B4FF4915061D6B9C667F299A
   8%2526source%253Dyoutube%2526expire%253D1335995606%2526key%253Dyt1%25
   26ipbits%253D8%2526factor%253D1.25%2526cp%253DU0hSSllUU19FUENOMl9RTld
   HOjVTUG1BQ0NRaVll%2526id%253D6daeee6759282265%26quality%3Dsmall

[don't use these links, the key value depends upon your IP address, you'll need to do this stuff yourself]


Now we've got ourselves a URL, it is time to perform some conversions to get it to work.
Search and replace as follows:

Search for	Replace with
%253F	?
%253D	=
%2526	&
%25252C	,
%26	&
%3D	=
%253A	:
%252F	/
%2526	&
&2C	,

[/size]

Yes, %2526 (&) is listed twice. This is intentional.


This will leave you with something that looks like a URL, in the following format:
   http://o-o.preferred.orange-par1.v2.lscache1.c.youtube.com/videoplayb
   ack?upn=nRoni3Eqzu4&sparams=algorithm,burst,cp,factor,id,ip,ipbits,it
   ag,source,upn,expire&fexp=909903,901604&algorithm=throttle-factor&ita
   g=5&ip=90.0.0.0&burst=40&sver=3&signature=13C8AE4B6228BD8B7E4BD68EBD7
   1DCEC7B5951D0.C326D73C133C465B4FF4915061D6B9C667F299A8&source=youtube
   &expire=1335995606&key=yt1&ipbits=8&factor=1.25&cp=U0hSSllUU19FUENOMl
   9RTldHOjVTUG1BQ0NRaVll&id=6daeee6759282265&quality=small


Toss this to your browser (or fetch into the OSD) and you'll see Tiny Alice running up the driveway with me. That old fashioned token stuff is no longer required. Probably half of the above isn't required either, but go with it...



There you go. How to turn a YouTube URL into a playable video.


Best wishes,

Rick.

[NZavaloff]

this I already know, and I wrote code to implement it some time ago.

[heyrick]

NZavaloff - thanks to Google's increasing lameness with searching, I found it easier to get stuck in and examine the data than attempt to look for a solution by others. I then made my posting because you will note that it is a description, not code. Back when I was looking to how YouTube linking worked originally, I found a lot of examples, like Perl code or stuff with a lot of baggage for special options and/or OS peculiarities. Thus, again, I decided a worked guide that you can do with nothing more than Notepad and a browser would be best. Code is largely irrelevant without understanding.

So... you've written code to do this? For the OSD? Where? God knows the built-in YouTube client is prehistoric...


Best wishes,

Rick.