errors booting new kernel

When I boot my new kernel at first it boots fine... then it prints many
(like couple of hundred) of such lines:

...
Empty flash at 0x00ea7ffc ends at 0x00ea8000
Empty flash at 0x00eb3ffc ends at 0x00eb4000
Empty flash at 0x00eb7ffc ends at 0x00eb8000
Empty flash at 0x00ec3ffc ends at 0x00ec4000
...

And at the end it says:
VFS: Mounted root (jffs2 filesystem) on device 31:3.
Freeing init memory: 124K
Kernel panic - not syncing: No init found.  Try passing init= option to
kernel. See Linux Documentation/init.txt for guidance.
[<c002cb7c>] (unwind_backtrace+0x0/0xec) from [<c02d7e24>]
(panic+0x54/0xdc)
[<c02d7e24>] (panic+0x54/0xdc) from [<c002764c>] (init_post+0x9c/0xbc)
[<c002764c>] (init_post+0x9c/0xbc) from [<c0008478>]
(kernel_init+0x114/0x150)
[<c0008478>] (kernel_init+0x114/0x150) from [<c0028a4c>]
(kernel_thread_exit+0x0/0x8)

Is the "empty flash" thing something to worry about? The rootfs partition
is for sure bigger than the jffs2 image that I wrote to it... 
Can I fix this somehow?

Also how do I pass the init= option to the kernel?

regards,
-- 
Luke

I added init=/sbin/init to bootargs but it still says:
Failed to execute /sbin/init.  Attempting defaults...
Kernel panic - not syncing: No init found.  Try passing init= option to
kernel. See Linux Documentation/init.txt for guidance.
[<c002cb7c>] (unwind_backtrace+0x0/0xec) from [<c02d7e24>]
(panic+0x54/0xdc)
[<c02d7e24>] (panic+0x54/0xdc) from [<c002764c>] (init_post+0x9c/0xbc)
[<c002764c>] (init_post+0x9c/0xbc) from [<c0008478>]
(kernel_init+0x114/0x150)
[<c0008478>] (kernel_init+0x114/0x150) from [<c0028a4c>]
(kernel_thread_exit+0x0/0x8)

Yes, I think it is something to worry about.

However, I am struggling to keep up with where you are at. Maybe, if you
describe your current setup.

IE Supervivi in NOR, u-boot in NAND, maybe reference the tutorial I posted.

I've finished the tutorial.
I have u-boot in both nor and nand.

MINI2440 # mtdparts 

device nand0 <mini2440-nand>, # parts = 4
 #: name                        size            offset          mask_flags
 0: u-boot              0x00040000      0x00000000      0
 1: u-boot_env          0x00020000      0x00040000      0
 2: kernel              0x00500000      0x00060000      0
 3: rootfs              0x3faa0000      0x00560000      0

active partition: nand0,0 - (u-boot) 0x00040000 @ 0x00000000

defaults:
mtdids  : nand0=mini2440-nand
mtdparts: <NULL>
MINI2440 #

I have download the kernel, and root and trying to boot the damn thing.

Here's the list of my env vars:

MINI2440 # printenv 
bootdelay=3
baudrate=115200
ethaddr=08:08:11:18:12:27
ipaddr=10.0.0.111
serverip=10.0.0.4
netmask=255.255.255.0
usbtty=cdc_acm
mini2440=mini2440=0tb
bootargs_base=console=ttySAC0,115200 noinitrd
bootargs_init=init=/sbin/init
root_nand=root=/dev/mtdblock3 rootfstype=jffs2
root_mmc=root=/dev/mmcblk0p2 rootdelay=2
root_nfs=/mnt/nfs
set_root_nfs=setenv root_nfs root=/dev/nfs rw
nfsroot=${serverip}:${root_nfs}
ifconfig_static=run setenv ifconfig
ip=${ipaddr}:${serverip}::${netmask}:mini2440:eth0
ifconfig_dhcp=run setenv ifconfig ip=dhcp
ifconfig=ip=dhcp
set_bootargs_mmc=setenv bootargs ${bootargs_base} ${bootargs_init}
${mini2440} ${root_mmc}
set_bootargs_nand=setenv bootargs ${bootargs_base} ${bootargs_init}
${mini2440} ${root_nand}
set_bootargs_nfs=run set_root_nfs; setenv bootargs ${bootargs_base}
${bootargs_init} ${mini2440} ${root_nfs} ${ifconfig}
mtdids=nand0=mini2440-nand
mtdparts=mtdparts=mini2440-nand:0x00040000(u-boot),0x00020000(u-boot_env),0x0050
0000(kernel),0x3faa0000(rootfs)
bootcmd=nboot.e kernel ; bootm
bootfile=x256l26359.jffs2
loadaddr=0x31000000
bootargs=root=/dev/mtdblock3 rootfstype=jffs2 console=ttySAC0,115200
init=/sbin/init rootwait=5
partition=nand0,0
mtddevnum=0
mtddevname=u-boot

Environment size: 1220/131068 bytes
MINI2440 #

Now I see there are errors in the nand after reset...

MINI2440 # reset&#65533;

U-Boot 1.3.2-mini2440 (Oct 30 2011 - 10:56:54)

I2C:   ready
DRAM:  64 MB
NOR Flash not found. Use hardware switch and 'flinit'
Flash:  0 kB
NAND:  1024 MiB
Found Environment offset in OOB..
USB:   S3C2410 USB Deviced
In:    serial
Out:   serial
Err:   serial
MAC: 08:08:11:18:12:27
Hit any key to stop autoboot:  0 

Loading from NAND 1GiB 3,3V 8-bit, offset 0x60000
   Image Name:   
   Created:      2011-10-30  22:50:34 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    2076216 Bytes =  2 MB
   Load Address: 30008000
   Entry Point:  30008000
reading NAND page at offset 0x25a800 failed
** Read error
Wrong Image Format for bootm command
ERROR: can't get kernel image!
MINI2440 #

I think I will do the process all over again, starting with scrubbing the
nand...

I suspected at times that you could not just jump in anywhere, but to go
back to the nand scrub stage. I must have gone through that procedure over
a hundred times before moving on.

Also I'm using tftp instead of reading from sd card (as in tutorial).
I hope this shouldn't make a difference?
It's much faster and easier this way...

Shit, I made it all over again, from scrubbing to the very end - same
thing: empty flash and kernel panic :(

Now I'd really use some help, because I'm running out of ideas.

I'm thinking of clearing the partition table. 
But I'd need a factory default. Do you have one for the 1GB nand?

No, mine is an old 64M model

I think my jffs2 is bad... I can't check it as those mounting scripts (from
the tutorial) don't work for me. It says the jffs2 filesytem is ukknown.
Probably because of a missing kernel module...

The mounting script in the Linux tutorial on this site works for me . . .
many times.

You run that script on your host machine, to check your JFFS2 filesystem.

I know, but it doesn't work for me as I said. Anywys... I'm recompiling my
kernel :)

Ah, tiredness finally caught up on you!

How are you making you JFFS2 filesystem?  Hopefully, not:

[mini2440-bootstrap] # mkfs.jffs2 –lqnp –e 16 –r target –o
output/my_rootfs.jffs2

"I know, but it doesn't work for me as I said."

Just checking, that you were running it on the host.  What is your host
machine running? There weren't any special applications required on Ubuntu
9.04 or 10.04LTS to mount the JFFS2 filesystem.

Did you use proper NAND parameters when making JFFS2 image? I mean -e, -s
an -p option ?

I'm making jffs2 like it is stated in the wiki:
mkfs.jffs2 –lqnp –e 16 –r target –o output/my_rootfs.jffs2

But now I see it must be wrong :)

Yes.  the 16 is for a 64M NAND flash.  You needed a different number here. 
From memory all flash sizes from 128M to 1GB use the same number.  Look at
"erase block size".

Yes, I've looked already :)
This explains the mounting problems. But anyways. I'm gonna try a new
kernel too.

Ok. problems, problems, problems... Nobody wants to help with the 2.6.38.8
compilation problems, so I'm still trying to boot of the 2.6.35.9 patched
by serg_io.
No I don't have the "empty flash" messages. The root is mounted and also
init is found,... but killed ;)
VFS: Mounted root (jffs2 filesystem) on device 31:3.
Freeing init memory: 132K
Kernel panic - not syncing: Attempted to kill init!
[<c002eef4>] (unwind_backtrace+0x0/0xec) from [<c0312654>]
(panic+0x5c/0xe4)
[<c0312654>] (panic+0x5c/0xe4) from [<c0041fcc>] (do_exit+0x64/0x684)
[<c0041fcc>] (do_exit+0x64/0x684) from [<c00426a8>]
(do_group_exit+0xbc/0xf4)
[<c00426a8>] (do_group_exit+0xbc/0xf4) from [<c004e204>]
(get_signal_to_deliver+0x394/0x3e8)
[<c004e204>] (get_signal_to_deliver+0x394/0x3e8) from [<c002c410>]
(do_notify_resume+0x58/0x65c)
[<c002c410>] (do_notify_resume+0x58/0x65c) from [<c002a0f0>]
(work_pending+0x20/0x24)

I made my jffs2 with -e 128, I hope this is good.
Maybe I'm missing some libraries and init simply wont start?
I used the libraries from
arm-2008q3/arm-none-linux-gnueabi/libc/armv4t/lib/* and put them to /lib -
I hope that is also ok (however wiki tells me tu put them in the jffs root,
but "then Kernel panic - not syncing: No init found." So I don't know which
is worse).

I bet I need just a little tweak, addition link or library, mkfs.jffs2
setting change to make it work :/ HELP please

regards,

Ok. One step further... It was the filesytem as I suspected.
I built the whole thing using buildroot (a tool from busybox site) and the
linux starts... well I think it starts because the console ouput gets cut
as soon as it starts loading. 
This is what I get:

VFS: Mounted root (jffs2 filesystem) on device 31:3.
Freeing init memory: 132K
Starting logging: OK
Starting network...
Starting input-event-daemon: done
input-event-daemon: daemon(): No such device

And then the serial connection gets offline.
Now what? :)

Now its time to add the "init=/sbin/init" again. To omit all userland
guessing.

It's there afaik:

MINI2440 # printenv 
bootargs=root=/dev/mtdblock3 rootfstype=jffs2 console=ttySAC0,115200
bootdelay=3
baudrate=115200
ethaddr=08:08:11:18:12:27
usbtty=cdc_acm
mini2440=mini2440=0tb
bootargs_base=console=ttySAC0,115200 noinitrd
bootargs_init=init=/sbin/init
root_nand=root=/dev/mtdblock3 rootfstype=jffs2
root_mmc=root=/dev/mmcblk0p2 rootdelay=2
root_nfs=/mnt/nfs
set_root_nfs=setenv root_nfs root=/dev/nfs rw
nfsroot=${serverip}:${root_nfs}
ifconfig_static=run setenv ifconfig
ip=${ipaddr}:${serverip}::${netmask}:mini2440:eth0
ifconfig_dhcp=run setenv ifconfig ip=dhcp
ifconfig=ip=dhcp
set_bootargs_mmc=setenv bootargs ${bootargs_base} ${bootargs_init}
${mini2440} ${root_mmc}
set_bootargs_nand=setenv bootargs ${bootargs_base} ${bootargs_init}
${mini2440} ${root_nand}
set_bootargs_nfs=run set_root_nfs; setenv bootargs ${bootargs_base}
${bootargs_init} ${mini2440} ${root_nfs} ${ifconfig}
mtdids=nand0=mini2440-nand
bootfile=u-boot.bin
loadaddr=0x31000000
filesize=3A814
fileaddr=31000000
netmask=255.255.255.0
ipaddr=10.0.0.111
serverip=10.0.0.4
mtdparts=mtdparts=mini2440-nand:0x00040000(u-boot),0x00020000(u-boot_env),0x0050
0000(kernel),0x3faa0000(rootfs)
bootcmd=nboot.e kernel ; bootm
partition=nand0,0
mtddevnum=0
mtddevname=u-boot

Environment size: 1220/131068 bytes

The kernel reports what command line content it received, like this one:

[...]
Kernel command line: console=ttyS2,115200n8 noinitrd rw root=/dev/nfs
nfsroot=/home/jb/work/ptx/bsps/mybsp/platform-myplatform/root,nolock,tcp
ip=dhcp debug earlyprintk
[...]

You should see in your kernel log, if the "init=" is set correctly.

good to know, thank you. I actually didn't have it in the kernel command
line. I added it and it doesn't change the unwanted behavior. 
However something new appeared (CRC error):

...
Kernel command line: init=/sbin/init root=/dev/mtdblock3 rootfstype=jffs2
console=ttySAC0,115200
...
s3c-rtc s3c2410-rtc: setting system clock to 2000-05-02 15:50:45 UTC
(957282645)
jffs2_scan_inode_node(): CRC failed on node at 0x38816efd4: Read
0xffffffff, calculated 0x0d611447
VFS: Mounted root (jffs2 filesystem) on device 31:3.
Freeing init memory: 132K
Starting logging: OK
Starting network...

It wasn't there before for sure.

Now I have more of it, lol
Empty flash at 0x0b2c8584 ends at 0x0b2c8800
jffs2_scan_inode_node(): CRC failed on node at 0x38816fd4: Read 0xffffffff,
calculated 0x0d611447
Empty flash at 0x38817728 ends at 0x38817800

Badly rebuild bad block table? Or my NAND starts to break?

In my many hours of messing around with the own JFFS2 filesystems I would
generally end up with this warning.  Is seems that the root filesystem
throws up this message the first time you change anything after a fresh
boot.

Ignoring this message seemed to be a safe thing to do and no one could
offer any help on my concern.

Try searching <CRC failed on node at> on this site and Google.

I did not get the Empty flash message though.  I'd suggest that needs
sorting.

NAND breaking down?  You have been working this unit pretty hard, but I
doubt it.

Bad  block table?  That's why you need to do a nand scrub each time with
u-boot (correct me if I am wrong), which for your 1GB machine takes
alooooong time.

It takes about 11 minutes. But it is dangerous. As far as I understand it
the nand comes with a preprogrammed map of factory bad blocks. Scrubbing
removes this map copying it to memory and then recreating it. If something
goes wrong in the process you lose this precious information and can have
weird memory errors from then on... 
If someone knows better please correct me.

Anyways what should I try now to see the friggen shell on my console? :)

I already have set the getty in inittab accordingly:
ttySAC0::respawn:/sbin/getty -L ttySAC0 115200 vt102 # GENERIC_SERIAL

before it was ttyS0 also vt100 - no difference

I can write in the console but nothing happens, nothing is desplayed
(except for what I'm writing).

User root has shell /bin/sh which is present (the file)
/etc/passwd entry for root:
root:x:0:0:root:/root:/bin/sh

so no password.

What else can I do/check?

There's this file used by buildroot that is used to build /dev/* when
static dev is selected: device_table_dev.txt

It has a list of all created devices including ttySAC
Maybe ttySAC has bad majon/minor numbers? How can check that?

Buildroot's howto http://buildroot.uclibc.org/buildroot.html#faq) says it's
enough to set the serial port in the buildroot configuration. Symptoms are
exactly the same as mine, but the soltion doesn't wokr :(

So close... yet so far :)

Yup, every time you read something about scrubbing NAND you are given these
"dire" warnings.

And expressing concerns in this area as well was met with silence.

I scrubbed the NAND probably a hundred or more times with Bao's tutorial
and the unit still works.

Perhaps, what I need to know is WHY DO YOU (apparently) HAVE TO SCRUB THE
NAND? I understand that the BBT is modified whenever bad blocks are
encountered in the future.

Now, you got me thinking about what really happens when running Barebox in
the Pengutronix BSP, as it seems to set things up a lot quicker.

Your console problem:
- have you enabled virtual console (?) in the kernel

I'll fish out the details.

Device Drivers  --->
  Character devices  --->
    [*] Virtual terminal
    [*]   Enable character translations in console
    [*]   Support for console on virtual terminal
 
  Graphics support  --->
[*] Support for frame buffer devices  --->
<*> S3C2410 LCD framebuffer support       

    Console display driver support  ---> 
      <*> Framebuffer Console support
      [*] Map the console to the primary display device
      [*] Select compiled-in fonts 
      [*] VGA 8x8 font

Maybe ttySAC has bad majon/minor numbers? How can check that?

/dev/ttySAC* and ls -la

Ah, you mean when you enter a command you see the test displayed but that
on hitting the enter key nothing happens after that.

Don't know why.

Where? On localhost (host computer)? I don't have ttySAC. On mini? I don't
have shell! :)

Anyways I changed init=/sbin/init to init=/bin/sh.
I get a shell, but I don't think it's the way to go...

Freeing init memory: 132K
/bin/sh: can't access tty; job control turned off
/ #

Are you able to run something like minicom on the host and talk to
mini2440, ie navigate the root directory?

ttySAC0, ttySAC1 and ttySAC2 should be on the mini2440.

I think I might have lost the "top-level" view of what you are trying to
do.  I never used buildroot.  And after playing about this device for about
2 years I am now only using the Pengutronix BSP.

Off to work I go.

Yes, I can do cd,ls...
I wonder what are the drawbacks of using /bin/sh as init...

What do you see if you cd to /dev/ttySAC* and ls -la  ??

/ # ls -al /dev/ttySAC*
-rw--w----    1 root     root            59 May  2 11:20 /dev/ttySAC0
-rw-r--r--    1 root     root             0 Nov  1  2011 /dev/ttySAC1
-rw-r--r--    1 root     root             0 Nov  1  2011 /dev/ttySAC2
-rw-r--r--    1 root     root             0 Nov  1  2011 /dev/ttySAC3

I haven't got access to my mini2440 at work, but I don't see the major and
minor numbers in your clip.  I thought they were suppose to be before the
date.

Are you running mdev or udev?

neither, static dev - as I wrote before

Sorry, sometimes I don't pick up important details.  I have no static dev
experience.

Talking about experience . . . you may have only started "Linux since last
Friday", but you do have significant other computer/software experience?

Well, yeah, I've been around for a "few" years ;)

Why are you using static device nodes???? If you do not want to use udev
(as it slow down system's start up) you can use the devtmpfs instead. Using
static device nodes is error prone.

"I wonder what are the drawbacks of using /bin/sh as init..." Hit CTRL-C
and you will see it. ;-)

"jffs2_scan_inode_node(): CRC failed on node at 0x38816efd4: [...]". If you
erase the flash and write a new JFFS2 image into it, and you mount this
filesystem the first time, JFFS2 scans the whole flash in the background.
You can see it, if you configure your kernel to flash one of the LEDs
(refer "LED trigger") if an NAND access occurs. If you disturbing this scan
by hitting the RESET button, because you are impatient, such messages can
occure the next time you boot the system. Mostly harmless and you can
ignore them (AFAIK).

ctrl-d u mean :)
well... then I have to recompile the kernel, as it doesn't include devtmpfs
:/
Anywyas... with /bin/sh as init i can mount proc but I can't run some of
the xenomai binaries... 

/usr/xenomai/bin # sh ./xeno latency
./xeno: exec: line 15: /usr/xenomai/bin/latency: not found
/usr/xenomai/bin # ls -al /usr/xenomai/bin/latency
-rwxr-xr-x    1 root     root         29528 Nov  1  2011
/usr/xenomai/bin/latency

I've compiled the kernel with devtmpfs and changed static dec to devtmpfs.
Now the boot process looks like this: 
...
Starting network...
can't open /dev/ttyS0: No such file or directory
can't open /dev/ttyS0: No such file or directory
...

or if I change to /dev/ttySAC0 then you can substitute in the above
lines...

This is bad... init=/bin/sh

/usr/xenomai/bin # ls -al /dev/ttyS*
ls: /dev/ttyS*: No such file or directory

???

Haha, changed the inittab entry to console...
::respawn:/sbin/getty -L console 115200 vt100 # GENERIC_SERIAL

and I get the login prompt!

But ;)
I don't know how to login :)

...
Starting network...

Welcome to Buildroot
buildroot login: root
Login incorrect
buildroot login:

So what next? :)

root

oops that's for the Pengutronix BSP.

Someone did ask that same question on here in the last year or so.

Search <buildroor login>?

<buildroot login>!

Default works, and now I added console to /etc/securetty.
Soon we'll see :)

Eh... I hate this. When I changed ttySAC0 to console on the device it
worked after reboot.
Now when I changed it on the host and made fs again and put it back on the
mini2440 it doesn't work anymore...

... becasue I made a typo :(
Eh, time to sleep soon...

I thought default didn't work, as per your last post.  This is happening
too fast!  Spelling mistakes and all.

it works! Almost ;)
Welcome to Buildroot
buildroot login: root
-sh: can't access tty; job control turned off
#

I'd love to get rid of the waring/error message...
And now it's time to make the xenomai work ;)

Man, you managed to stuff in two more posts before I could respond!!

-sh: can't access tty; job control turned off

Another message I think you can ignore.

Are you going to write a tutorial when you xenomai working on the mini2440
:)

insert "get" before xenomai!

Maybe some day... ;)
Now it still seems impossible to me :)
I get too many nasty errors ... and help from nowhere - I'm a pioneer ;)

I hope you are referring to xenomai.

Is there not a forum associated with xenomai?

Tomorrow is another day.

Sure I'm referining to xenomai. With mini I get help for which - thank you
all :) About xenomai I'm on a mail list - but they're not as fast as you ;)

Those were an unusual 2 or 3 days.  Because I am in NZ I am used to posting
a question and maybe hearing back 6-12 hours later.

Now that you have done things the hard way, have you had a look at the
Pengutronix BSP for the mini2440?

Yes, I've taken their kernel patches to try to build xenomai 2.6.0 on them.
 But the kernel won't compile yet.

I didn't read the complete conversation. But i have also faced this crash
when i was booting filesystem from Nand flash. This is because, When you
boot first time, kernel and fs boots without any issue. After booting, it
will have a sync with the fs partition in nandflash. Read and write will be
happening frequently. If you reset the board this sync is lost and it will
corrupt the filesystem, so second it will not boot. Instead of reset try
rebooting with the reboot command. (reboot -f also might not work). reboot
will do proper operation with the fs so second time it should boot without
any issue. Try with the reboot command still if you face this issue then
its something else.

changed the inittab entry to console...
::respawn:/sbin/getty -L console 115200 vt100 # GENERIC_SERIAL

i am not getting the serial console this error.

VFS: Mounted root (yaffs filesystem) on device 31:3.
Freeing init memory: 188K
Warning: unable to open an initial console.
request_module: runaway loop modprobe binfmt-464c
request_module: runaway loop modprobe binfmt-464c
request_module: runaway loop modprobe binfmt-464c
request_module: runaway loop modprobe binfmt-464c
request_module: runaway loop modprobe binfmt-464c

pls any one help me inittab pls send the my mail

Thanks