Setting up "netbooting" on the OSD...

[pfft2001]

I propose that we work together to develop a method for netbooting the OSD.

In particular, one that does not depend on downloading and rebuilding the
entire firmware.  Also, I want to create a setup that can be used in the
context of KTS (see below).

Advantages to netbooting include:

    1) Reliability over relying on the onboard firmware.  As you know, I
   have had problems booting one of my OSDs lately.  I'd like to have
   an alternative.
    2) Making things read/write.  Since the netboot works off of NFS-mounted
   filesystems, these can be read-write, so there's no problem changing
   things.  The fact that everything is (well, most things are)
   read-only in the OSD has always been a hindrance to hacking it.

Now, I am aware of the Wiki pages describing how to setup netbooting, but
I'm not happy with them.  I won't say that they are wrong, but they don't
have the right orientation for me.  The Wiki pages are oriented towards two
assumptions, both of which I am trying to get away from:

    1) That you have a build environment and have downloaded (successfully)
   and compiled (successfully!) the entire OSD firmware.
    2) That you're not familiar with netbooting in general.  I.e., the Wiki
   article spends a great deal of time talking about setting up all the
   niggly little details of netbooting (which, don't get me wrong, are
   tricky to setup, so if you haven't done it before, you really need
   the context and level of detail that these pages provide).  But note
   that most of it (setting up netbooting) isn't really specific to the
   OSD.

It is also true (IMHO, of course) that the Wiki presentation isn't
"exportable" - i.e., it isn't really "cookbook".  It seems a bit rambling;
it reads more like the textbook for a semester-length class in "How to work
on the OSD firmware", than like a simple cookbook recipe for setting up
netbooting.  Note that one of the consequences of assumption #1 above is
that they don't explain how it works (in the context of comparing it to
other netbooting setups), since many of these details are "baked in" when
you build the firmware.  For example, I didn't see any mention in there for
when/where/how the names of the NFS shares are determined.  See my list
below for the details that I seek.

   *   *   *   *

Now, my situation is that I have been using Knoppix (You can google it; I
don't have a URL at hand) for about a decade now and the thing that first
attracted me to Knoppix is the "Knoppix Terminal Server" (KTS).  This is a
really nice, self-contained, soup-to-nuts, implemention of a netbooting
setup.  It is implemented as a single, ungodly-complicated-but-it's-all-there,
shell script that sets up all the files, starts all the needed servers and
processes, and generally takes care of all the niggly little details
involved in setting up netbooting.

I have a Knoppix Terminal Server running on my network all the time.  One of
the really nice things about it is that a single server setup can serve any
number of OS configurations.  I currently use it to boot several different
Linux configurations as well as a few DOS (Yes, good old fashioned DOS!)
setups.

In KTS, there are 4 components that make up a boot entry (for Linux):

    1) A kernel (e.g., a vmlinuz file)
    2) An initial RamDisk (initrd).  This is usually packaged as a file
   called "miniroot.gz".  The initrd is basically a floppy disk image
   of a tiny Linux whose primary job is to NFS mount the "real" Linux.
   The key thing here is the "linuxrc" file.
    3) One or more NFS-mountable shares that contain the rest of the OS.
    4) An entry in the "pxelinux.cfg" file, that ties it all together.

Having said all this, I think that something close to this (see list below)
should work for the OSD.  But I would like comments and input from others
before I start working on it.

    1) Get a kernel - i.e., the uImage file.  I cannot get this from the OSD
   itself, since it doesn't exist as a file in the filesystem.
    2) Use tar to tar up the root and extended-area filesystems from my OSD.
   Then, untar these into the appropriately named directories on my
   server.  Put the appropriate entries into the /etc/exports file.
    3) Create the initrd (and I am totally clueless on when/where/how this
   file or equivalent functionality exists in the OSD world)
    4) Set up the pxelinux.cfg file line.  Again, I'm clueless on what to
   put here.

[heyrick]

I am working on an unpacker for the UPK files. Soon...?

Yes, it would be nice to have a step by step guide that doesn't make lots of assumptions. I may look to seeing if a Windows version is possible, as it would make dev work easier, though I won't be running netboot all the time (when I'm at work, the OSD is often the only thing on the LAN that is powered up).


One suggestion I can think of is to tweak the OSD's boot process to 'ping' the (predefined) netboot server. If it is there, attempt to pull uImage, and if not, boot as normal from Flash. This would give us the best of both worlds with no tweaks to the OSD, just run the server and reboot the OSD. When dev done, stop the server and reboot the OSD...


Best wishes,

Rick.

PS: I have had an offer of a "bricked" OSD. If it turns up, it will be an interesting project to recover it. I am hoping pushing U-Boot into the Flash would be sufficient? This is something that probably needs to be covered in our dev-docs.

[greyback]

I quite thought the wiki does a good job of explaining setting up the uBoot variables needed to netboot successfully.

Basically for netboot, you need a server with 2 abilities:
1. tftp - the OSD will use the tftp protocol to download the kernel image (uImage). You need a tftp-server.
2. nfs - Once the kernel is booted, it needs to mount a filesystem & call init. nfs does the job.

There is no initrd, and no pxelinux.cfg. It's even simpler.

The paths given in the guide are just a arbitrary choice, you can do as you like. As long as the details in uBoot match those on your server.

Just like Rick, I can supply a uImage file and rootfs which you can use, if you'd like.

Rick: I'm unsure if the OSD's uBoot shell install is capable of if/then/else stuff. It's not very smart. And I've no idea how to flash a correct  uBoot - and there may be data which is run even earlier, I've heard of a "UBL" - I hope Google is your friend.
-G
-G

[pfft2001]

I quite thought the wiki does a good job of explaining setting up the uBoot variables needed to netboot successfully.

Yes.  I looked at it again.  It is there.

Basically for netboot, you need a server with 2 abilities:
1. tftp - the OSD will use the tftp protocol to download the kernel image (uImage). You need a tftp-server.
2. nfs - Once the kernel is booted, it needs to mount a filesystem & call init. nfs does the job.

There is no initrd, and no pxelinux.cfg. It's even simpler.

To the original question posed in this thread, namely: Can it work with a KTS server?
the answer seems to be "no and yes".  I.e.,  it can't work as an entry in the pxelinux.cfg/default file, because pxelinux (Ed: part of the "syslinux" package) is itself x86 code.  I.e., pxelinux (running on x86) can only boot an x86 OS.

But, since KTS includes setting up a working tftp server, that part should still work.  It looks to me like it will work OK as long as I don't use DHCP - since it is the KTS DHCP server that serves up the x86 code running pxelinux.  And this is fine with me, because, for a variety of other reasons, I don't think it is a good idea to configure the OSDs with DHCP, and in fact all of my units are configured with static IP.

I'm still curious, though, as to which part of the OSD booting process performs the task of mounting the NFS shares - i.e., the part that I think of as being performed by "initrd" (in the KTS context).  Can you answer this question for me?

Also, can you confirm (or deny) that the OSD netbooting process is a "PXE" booting process?

The paths given in the guide are just a arbitrary choice, you can do as you like. As long as the details in uBoot match those on your server.

Just like Rick, I can supply a uImage file and rootfs which you can use, if you'd like.

Right, but I really need the OSDng kernel (because it has 2 needed features - see below for a list).  I'd really like to have Rick's "Un-UPK" utility - that is, a way to get it out of my OSDng UPK file.  But, if someone wants to send me the OSDng uImage file, that'd be fine too.

Re: the 2 filesystems that I need (the rootfs and the "extended area"), can I just get them from my existing system (by tarring them up, as I suggested in the original post) ?  (If not, then I guess someone should send them to me...)

Footnote: Reasons for using OSDng: 1) "mushy keys fix", so that x11vnc works and 2) kernel support for "loop" filesystems.  Both of these fixes are "must-haves" for me now, in using the OSD.

[heyrick]

And this is fine with me, because, for a variety of other reasons, I don't think it is a good idea to configure the OSDs with DHCP, and in fact all of my units are configured with static IP.

Why? Mine is DHCP, with a 'reserved' IP in the router so it will get the same one. For me, DHCP is necessary as it lives, normally, on the bedroom LAN, but for external tests, it connects to the LiveBox. The LAN is 192.168.0.12 and the LiveBox is 192.168.1.12...

I'm still curious, though, as to which part of the OSD booting process performs the task of mounting the NFS shares - i.e., the part that I think of as being performed by "initrd" (in the KTS context).  Can you answer this question for me?

I guess the question is does U-Boot do this, or does the loaded kernel?

I'd really like to have Rick's "Un-UPK" utility

Hopefully I will have it ready sometime this weekend. I didn't work on it last weekend (sorry!), too damn tired.

Footnote: Reasons for using OSDng: 1) "mushy keys fix", so that x11vnc works and 2) kernel support for "loop" filesystems.  Both of these fixes are "must-haves" for me now, in using the OSD.

We must consider the possibility of having to re-patch the kernel, so...

Mushy keys - what is this?

Loop filesystems - I guess this is a way to make a file a read/write filesystem, yes?

How would we go about adding this to the kernel?


Best wishes,

Rick.

[pfft2001]

Footnote: Reasons for using OSDng: 1) "mushy keys fix", so that x11vnc works and 2) kernel support for "loop" filesystems.  Both of these fixes are "must-haves" for me now, in using the OSD.

We must consider the possibility of having to re-patch the kernel, so...

Mushy keys - what is this?

Loop filesystems - I guess this is a way to make a file a read/write filesystem, yes?

How would we go about adding this to the kernel?


Best wishes,

Rick.

1) "mushy keys".  First, I didn't coin this term; it was in standard use by others on this board whenever this topic has been discussed.  It refers to the fact that there is an VNC server for the OSD, that someone ported to the OSD and made publicly available.  Unfortunately, it's not usable with either stock Torfu or Arizona firmware, because when you press a key (from the client) side, it generates multiple keystrokes on the server.  Someone (again, not me) termed this "the mushy keys" problem - since its like a single keystroke generates a mush of keys.

The OSDng kernel fixes this, so that x11vnc is now usable.  And the fix is referred to as "the mushy keys fix".

2) Yes, the "loop" device is a way of treating a file as a filesystem.  This is standard Linux; you can Google it and read all about it.  It has nothing per se to do with the OSD.  But for some reason, the stock Arizona kernel does not have it compiled in.  "Full blown OSDng" (i.e., what you are running, but not me) depends on the loop functionality; the rootfs and the "extended area" are both setup as loop filesystems.

As for your question: How do we do this?  The best answer is to find the OSDng guy and use his work.  Failing that...

1) "mushy keys".  Not sure.  Since this is OSD-specific, I don't know where it would come from.

2) "loop".  This is standard Linux, so it is probably just a matter of re-compiling the kernel with this option turned on.  Since you have the sources working, you should take a look at it.

[heyrick]

Compile with this option turned on?

Okay... Where would I find options?


Best wishes,

Rick.

[pfft2001]

Compile with this option turned on?

Okay... Where would I find options?


Best wishes,

Rick.

How much do you know about Linux kernel compilation?  Have you ever done it?

I've done it, but not in a long, long time.  When Linux first came out (almost 2 decades ago), it was a common and fairly simple operation.   Anyone who used Linux back in those days was expected to compile their own kernel, so we did.

But basically, you do some version of "make config" and it asks you a slew of questions about which options you want compiled in.  One of them (one of the questions) should be about the loop filesystem.  "make config" creates a configuration file that is used by the actual kernel compilation to tell it what to do.  When you do "make", it uses this file to compile the new kernel.

P.S.  I'm assuming that when Neuros builds a kernel for the OSD, they turn off any option that isn't necessary, to save disk space and memory.  So that's why the loop is missing from the stock kernels.

[heyrick]

I know about the RISC OS kernel... ;-)

Linux? Next to nothing. Yes, yonks ago I had a book with Slackware on a CD. Linux loaded up on an old 486. Cool, let's play, I thought. Opened the book. Chapter one, sentence one... First, build your kernel. WTF? It has already loaded. Maybe sub optimal, but how about we start with an introduction that is useful? I closed the book, ejected the CD... If it wasn't for the OSD, I would still have minimal exposure to Linux. It isn't that Windows is better (ha ha ha), it just seems that if there is an easy way and a hard way, Unix mentality takes the hard way. But, hey, Microsoft is planning an ARM port of Windows. I'd say the boat hasn't just been missed, it has arrived at its destination!

Lots of options? Yes, I saw that when building the kernel. I'll need to see where the make script gets its responses from...

Best wishes,

Rick.

[heyrick]

Hi,

Built a new kernel, logging the output. Lots of options, but I am not sure where the answers come from as I would be interested in ADFS partition support (personal reasons) and to add gzip and diff to busybox.

I did note:
  Support loopback mounts (FEATURE_MOUNT_LOOP) [Y/n/?] y
Is this what is needed, or does it require this too:
losetup (LOSETUP) [N/y/?] n
?

The full log of the bsp build (with lots of options listed), is at http://www.heyrick.co.uk/osd/build-bsp.txt (242KiB).

I have put up this kernel. The original is uImageA and this is uImageB. It is, oddly enough, 4 bytes smaller. Well, if anybody is brave, give 'em a whirl!
http://www.heyrick.co.uk/osd/


Best wishes,

Rick.

[pfft2001]

Hi,

Built a new kernel, logging the output. Lots of options, but I am not sure where the answers come from as I would be interested in ADFS partition support (personal reasons) and to add gzip and diff to busybox.

I did note:
  Support loopback mounts (FEATURE_MOUNT_LOOP) [Y/n/?] y
Is this what is needed, or does it require this too:
losetup (LOSETUP) [N/y/?] n
?

The full log of the bsp build (with lots of options listed), is at http://www.heyrick.co.uk/osd/build-bsp.txt (242KiB).

I think the relevant part (of your log file) is:

*
* Block devices
*
Loopback device support (BLK_DEV_LOOP) [Y/n/m/?] y

The parts that you quoted had to do with the applets of busybox, not the kernel itself.  Of course, you do need both of those things available (presumably, in busybox) in order for the loop devices to be usable (those things being the "losetup" program and the ability for "mount" to work with loop devices).

I have put up this kernel. The original is uImageA and this is uImageB. It is, oddly enough, 4 bytes smaller. Well, if anybody is brave, give 'em a whirl!
http://www.heyrick.co.uk/osd/

OK - that's a good thing, but I still can't proceed with netbooting until someone answers my questions about the filesystems (rootfs and "extended area").

[heyrick]

OK - that's a good thing, but I still can't proceed with netbooting until someone answers my questions about the filesystems (rootfs and "extended area").

There's a rootfs on my site (may our may not work ) as cramfs or tar.... You're on your own here, though I'd be inclined to try out the method in the wiki. There seems to be no mention of PXE, so maybe U-Boot just loads up the uImage and there's no extra complication?

Sorry I can't be more helpful, my OSD gets regular use as intended.

I think the mushy keys might be an issue of denounce? I'll need to poke around the IR driver, see if I can spot anything. <thinks> As I'm running OSDng, I might be able to paste the 'broken' Arizona driver in, and see how it behaves.
Is there a quick and simple way to unload/load kernel modules via telnet?


Best wishes,

Rick.

[heyrick]

First release, as promised.
You will find a better version two in its own topic.

But NOTE WELL...

• Has NOT been tested, Lightly tested by comparing some files written to UPK, and that which was extracted.
• Cannot extract cramfs files (they're split, and part two recreates the file). I'll look to fix this asap.
• Doesn't understand Torfu UPKs (no signature). Actually, to be honest, I think the UPK format is a klutzy mess...
• It is real slow. It byte-copies. :'( Sorry. I had little sleep last night, got up early today, and it's nearly half three in the morning. I feel half-dead, so you're probably safer with a lame byte copy than a nice chunk-copy routine that, oops, has a side effect of translating everything into Finnish (where messages like "No storage" would end up looking like "wow, että olisi eeppinen epäonnistua"!).
• Probably some other stuff...

You have here an executable for Windows (NT console), plus the source code (released under the EUPL). It is written to be portable... All the junk is for OpenWatcom. For other setups, just pull the single source file into a generic makefile and build it. Self contained, doesn't need any funny stuff. Uses C++ comments (the // style, so be careful with decades-old compilers!).


Um, that's it. I'm rambling now. Anyway, tread lightly, use at your own risk, etc etc.

pfft, I hope this at least gets you an Arizona uImage to work with.


Right. I'm going to crash now, my page tables need defragging. [exclamation icon here!]


Best wishes,

Rick.

Edit: I've made a nicer/sexier/better version two and precompiled it for Windows, x86 Ubuntu, the OSD (yes, really!), and RISC OS. It is described/available in its own forum topic. 

[pfft2001]

Thanks, Rick.  As you may have guessed, I didn't get around to doing any testing with Version 1, 'cause I just knew there'd be an updated version.  Looks like you got the necessary bells and whistles in place in this release.

I'll let you know my comments when I do get around to playing with it.  But the basic idea looks good, so, thanks again!

[heyrick]

'cause I just knew there'd be an updated version.

Well, my pride wouldn't let a treacle-slow byte copy hang around for long. Plus, it needed some extra testing.

There is one bug, the over-extends message has no parameter (why didn't OpenWatcom report this? Everything else did!). Well, that can wait for v0.03.

Looks like you got the necessary bells and whistles in place in this release.

As I said, I'd like to build up a (hopefully) useful little library of tools. I'm surprised, actually, we're this far down the line without a way of unpacking UPKs.

I'll let you know my comments when I do get around to playing with it.

I look forward to it.


Best wishes,

Rick.