gmane.os.netbsd.ports.sgimips

IP32 debugging kernel intr handler (crime/mace)

roop — 2013-03-19T10:11:59

Hi,

I'm digging into 47640 on sgimips (SGI O2 IP32) on netbsd-current
(6.99.17).

http://gnats.NetBSD.org/cgi-bin/query-pr-single.pl?number=47640

Background
-----------------
I'm basically starting out on kernel tinkering.

As explained in the gnats this SGI IP32 will hard lock at random point.
Otherwise it operates very nice and quickly. That was 6.0.1, same deal
on current. 
Also 47639, a random keyboard flash co-incides with a interrupt, may be
related.

What I want to do, but don't know if i can:
- get a debugger triggered at the right point, which means finding the
  right spot
- adding more debug around some other bits of kernel code, but not so to
  saturate the console
- breaking the kernel (linux magic sysreq style) and just getting it to
  dump it's stack at a keypress

The hunch was a glitch in the interrupt handling, so I've added
aprint_debug() liberally throughout dev/crime.c and mace/mace.c, as seen
in
https://gist.github.com/rooprob/5191131

The pattern I've discovered seems always "hang" after a complete
crime_intr()/mace_intr() loop, or immediately before the next one.
I'm completely open to the fact that thereofre this means the hang is
somewhere else.
There'll be hundreds of successful debug messages, occasionally a NULL
function pointer I've trapped with my code but it always end the same
way.

crime_intr()
  -> mace_intr()
     -> finish mace_intr
<- finish crime_intr

Some examples of my debug output:

XXX2013-RPF crime_intr(2111091192, 536936211, 1024)!
XXX2013-RPF <- !crime_ipending 0
XXX2013-RPF crime_intr(2152175152, 536936195, 1024)!
XXX2013-RPF <- !crime_ipending 0

XXX2013-RPF crime_intr(2110433472, 536936211, 1024)!
XXX2013-RPF -> mace_intr(256)
XXX2013-RPF -> mace_inst(256): processing non-IRQ 4
M0:256 M1:256 M2:256 M3:256 M4:256 M5:256 M6:256 M7:->(0x80074124)
M8:256 M9:256 M10:256 M11:256 M12:256 M13:256 M14:256 M15:256 M16:256
M17:256 M18:256 M19:256 M20:256 M21:256 M22:256 M23:256 M24:256 M25:256
M26:256 M27:256 M28:256 M29:256 M30:256 M31:256 done
XXX2013-RPF <- !crime_ipending 0

This last one, MACE idx 7 in maceintrtab fired.

Mar 18 23:22:03 mapleleaf /netbsd: XXX2013-RPF crime_intr(4255804, 536936211, 1024)!
Mar 18 23:22:03 mapleleaf /netbsd: XXX2013-RPF -> mace_intr(16)
Mar 18 23:22:03 mapleleaf /netbsd: XXX2013-RPF -> IRQ4 mace_intr(16) is IRQ 4!!! 
Mar 18 23:22:03 mapleleaf /netbsd: XXX2013-RPF -> IRQ4 mace_intr(16) completed mips3_ld() 
Mar 18 23:22:03 mapleleaf /netbsd: M4:0:16 M4:1:->(0x800d4348) M4:2:16 M4:3:16 M4:4:16 M4:5:16 M4:6:16 M4:7:16 M4:8:16 M4:9:16 M4:10:16 M4:11:16 M4:12:16 M4:13:16 M4:14:16 M4:15:16 M4:16:16 M4:17:16 M4:18:16 M4:19:16 M4:20:16 M4:21:16 M4:22:16 M4:23:16 M4:24:16 M4:25:16 M4:26:16 M4:27:16 M4:28:16 M4:29:16 M4:30:16 M4:31:16 done
Mar 18 23:22:03 mapleleaf /netbsd: XXX2013-RPF -> mace_inst(0): processing non-IRQ 4
Mar 18 23:22:03 mapleleaf /netbsd: M0:0 M1:0 M2:0 M3:0 M4:0 M5:0 M6:0 M7:0 M8:0 M9:0 M10:0 M11:0 M12:0 M13:0 M14:0 M15:0 M16:0 M17:0 M18:0 M19:0 M20:0 M21:0 M22:0 M23:0 M24:0 M25:0 M26:0 M27:0 M28:0 M29:0 M30:0 M31:0 done
Mar 18 23:22:03 mapleleaf /netbsd: XXX2013-RPF <- !crime_ipending 0


Causing a crash:
----------------------
With all my new debug spewing to the screen I can often get through a
good amount of atf-run before it will abruptly stop. It dies at
different places each time. If it's a subtle timing issue, it could be
masked by the new slowness of having to output to serial line, but I'm
not so sure about that.

Example output from rtf-run,
fs/ffs/t_setquota (84/498): 40 test cases
    set_be_1_both: [3.021925s] Passed.
... <snip>...
    set_default_le_1_both: [2.517445s] Passed.
    set_default_le_1_group: <died here>

Copying a new kernel from an NFS remote build server is my usual sure
fire way of getting it to hang quickly. This is why I'm thinking
interrupts. What doesn't gel with this idea is how long it can go - it's
interrupts and something else, perhaps.
What's interesting is that the intr handler stands up well against a
deluge of serial debug message output.

Where next?
----------------
Now I know how to make it crash, and now I've exposed the intr handling,
what should I look at next?

hunch#2: a missed interrupt ? kernel waiting forever?
hunch#3: a corrupted function (I tried to add sanity around that in my
patch where function !=NULL and managed to log a few, but to no avail)
hunch#4: something which happens outside intr handler space

I'm going to have to get the output-deluge down as it's too much for
syslog to capture, nor a serial line at 115200baud. The gfx console
happily keeps up, but I cannot copy and paste that to record the event. 

thanks for reading this far.

rob

Re: sgi O2 VICE and Nintendo64

Michael — 2013-03-05T13:59:45

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

On Mar 5, 2013, at 6:17 AM, roop wrote:


It's in -current Xorg as well. Using VICE there makes little sense  
since:
- - the MTE is much, much faster. Can't always use it but if we can it's  
running circles around everything else you can find in the O2.
- - the drawing engine supports everything we need in a console driver  
and basic X, I doubt VICE could beat it speed-wise doing simple,  
unaligned bit ops.
VICE is useful for more complex stuff that the drawing engine can't  
do, like video filtering and whatnot. Since it's a separate, fairly  
non-standard CPU ( a funky R3k-ish thing with a SIMD unit instead of  
an FPU ) that's kinda complicated though.
If anybody wants to give it a shot I have the manual.

have fun
Michael

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: GPGTools - http://gpgtools.org

iQEVAwUBUTX6UcpnzkX8Yg2nAQKVEQgAiF/9b7+y54PqhgL/XhMrnsSlflptRMIw
CFygXWKv107ysYhfCiST8l6lZ8j6frVQUCpOlSjP/fPla2/3yucT4kvUIluhCS0O
txA4Jjiii0/fkDbTDRO/Hc+9ti+mOUcIPEhGQ6vJf/bcyzM3FXcbnPCOB2ppxJp0
l5ojOes0QclaOkApUVWnfKDTS7S6lzqZ3yC3iePqFqnONHHrie7tyvJ7Jd/Zgzbg
ZNeN0dA9M1X2MEJ0d8kN2AUEyg4Rd6K6H+X6CXwd/kASewPg/e4G6ibbfVJWdSOU
wfNrdcnqalmV4EGK5DfDd6V9hsdkpc1sIqBSOCVWOEh6bVd9G36EzQ==
=W2dD
-----END PGP SIGNATURE-----

sgi O2 VICE and Nintendo64

roop — 2013-03-05T11:17:44

I had a memory burp: O2 VICE and Nintendo64 ; someone asking about N64 kernel; 
Brain righted itself n64 (32/64bit ABI) not N64 (Nintendo), or n32 or o32 subset.

Anyway, I randomly googled, refreshing my world view that the O2 is scattered
amongst the different projects;
- The hardware acceleration in VICE (derivative found in Nintendo64) was in
  Linux 2.5.x (http://www.linux-mips.org/~glaurung/), 
- Macallans excellent O2 2D acceleration work (didn't use VICE IIRC) X11 work
  in NetBSD (http://my.opera.com/Macallan/blog/2009/04/01/xorg-on-o2) 
  (Was this only XFree, or was carried over to Xorg?)
- OpenBSD's support for the O2 system generally seems more featureful and
  stable.
- Finally, back to IRIX, the guys at Nekochan have updated Firefox to 3
  (http://forums.nekochan.net/viewtopic.php?f=15&t=16727403) for IRIX and report
  faster, more stable behaviour than ff2.x (haven't tried it). 

But what's happening over in Nintendo world, and we find some nice research
into RSP from 2012/2013,

http://forum.pj64-emu.com/showthread.php?t=3398
https://github.com/cxd4/rcp/tree/master/rsp

Is O2 VICE and the Nintendo64 near enough this research useful?

Just posting if this is interesting.

rob

Re: PR 36158 (>256MB in O2)

Michael — 2013-03-03T14:32:29

Hello,

On Sun, 3 Mar 2013 09:21:09 +0100
Martin Husemann <martin< at >duskware.de> wrote:


Older MIPS CPUs ( not sure if it's all of them or just R5k and the like ) have some cache issues ( and had them for a long time, the last working kernel/userland I have is 5.99.17 vintage ) - corruption happens when data gets moved between processes. I don't see this on Loongson hardware although there's still corruption on NFS writes ( not sure if that's the same issue though )
Otherwise, N32 kernels with N32 userlands work more or less. I didn't try N64 on my O2 yet but I'd expect some trouble ( kseg vs. xphys, assumptions about LP32 etc. )

have fun
Michael

Re: PR 36158 (>256MB in O2)

Martin Husemann — 2013-03-03T08:21:09

Out of curiosity:
what is missing now for a N64 kernel? I had the impression other mips archs
do that now, so it can not be too much.

Martin
(I need to fix the rtc in my O2 and try)

Re: PR 36158 (>256MB in O2)

Stephen M. Rumble — 2013-03-03T04:05:22


I don't think there's been much activity on the sgimips port for a long time. You may be better off just using OpenBSD as Miod and company seem to be more active (and have the support you seek already). I think their sgi port is almost entirely a superset of ours by now.

In any event, wouldn't the library be much better off with a bunch of cheap atom machines? These SGIs are pretty ancient, and atoms would probably pay for themselves in power savings over a few years.

Steve

PR 36158 (>256MB in O2)

Kevin Day — 2013-03-02T23:47:15

Is anyone working on PR #36158 (NetBSD/sgimips does not properly detect the total amount of physical memory in an IP32 (O2) machine)?

I'm working with a library who wants to reuse a bunch of O2 machines as locked down kiosk web browsing systems, but I wasn't aware of this limitation until we got reasonably down this path. Looking at the diff for OpenBSD to add support it doesn't seem trivial, but I'm not sure if all of that is needed here. Any progress on this since the last PR update?

Re: documentation

folkert — 2013-02-13T09:48:37

This is great! I did not expect so much documentation to be available!


Yeah I decided this morning to go for an Indy emulation as there's more
documentation available for it.

Thanks!


Regards,

Folkert.

Re: SGI IP20 & 0xffffffffbfa00030

Stephen M. Rumble — 2013-02-12T22:37:27



[Oops, didn't reply to all.]

I think that's the memory controller (MC) chip accessed via kseg1. NetBSD calls the device "imc". Our imcreg.h header doesn't appear to contain any definition of the 0x30 offset, though.

The Indy docs should help you out there.

Steve

SGI IP20 & 0xffffffffbfa00030

folkert — 2013-02-12T20:37:29

Hi,

What on earth is located at address 0xffffffffbfa00030 on IP20 systems?
While tracing through the prom of such a system, I found it starts to
poke at that memory location. Can't find any docs on it.


regards

Folkert van Heusden

documentation

folkert — 2013-02-12T20:28:29

Hi,

I'm looking for any documentation on SGI hardware. E.g. Indy, Indigo,
etc.
This for an emulator I'm writing.


regards,

Folkert van Heusden

watches and high spirits you.

Freddy Newman — 2013-01-24T18:13:47

j t ckj pt hm
jfqkHI,
liwetealoffernqqyouulocotwonderfuliojxjrwatcheszvfev-PORTAL http://bit.ly/UNX44y mrxfsu vwxiq mubkeuq ccwjcitx sxmvpq
uctetydw xbnol jvvvns vwxlqygr oivar
mshlpmv zhrpj phobylku j uvc

word145324 — 2013-01-24T17:33:23


邮件事业部技

MIPS64 support?

Jerome Ibanes — 2012-10-07T23:14:21

It's my understanding that MIPS64 support was
added with the imminent release of NetBSD 6.0.

What are the implications of this for the sgimips
port? Would it ease the support for IP27, IP30
and IP35 machines?


Cheers,
J.

Re: bootp could not connect to server (no source for :.)

Pierre-Philipp Braun — 2012-06-30T19:13:28

David, Fabian,

There was something wrong with my terminal.  I've switched to another 
one and it works now.

About the bootp/network issue, I was missing the 'host' declaration in 
dhcpd.conf.  I though the IP could be simply attributed with the 
'subnet' declaration, but using fixed-address against the ethernet 
address, instead of the general 'range', helped a lot.  Maybe 
server-name="x.x.x.x"; too.  I didn't have that one, I only had 
next-server x.x.x.x.

I'm getting into sysinst, everything's fine.  But I just figured out 
there doesn't seem to be a harddrive in that O², according to what 
'hinv' says.  So I'll have to find relevant SCSI weiring and drives to 
connect to.

Thanks,
Pierre-Philipp

Re: bootp could not connect to server (no source for :.)

Fabian Wenk — 2012-06-29T14:01:24

Hello Pierre-Philipp

On 29.06.2012 14:07, Pierre-Philipp Braun wrote:

A long time ago I wrote the notes "How to install NetBSD 2.0_BETA 
on a SGI O2 with MIPS R10000 CPU" [1] down, maybe they are any 
help to you, it includes even the dhcp config needed to netboot.

   [1] http://www.wenks.ch/fabian/NetBSD-SGI-O2.txt


bye
Fabian

Re: bootp could not connect to server (no source for :.)

David Brownlee — 2012-06-29T12:24:59

Try with/without hardware flow control?


Do you see anything from the SGI on the server? (
tcpdump ether host 00:11:22:33:44:55)

bootp could not connect to server (no source for :.)

Pierre-Philipp Braun — 2012-06-29T12:07:38

Hi,

I've got two questions about an O² I recovered.  First, I've got 
troubles using the serial console.  I can see the text but it doesn't 
respond to inputs.  I need to press '5' to access the command monitor 
and nothing happens.  Maybe that's related to my terminal though, I'll 
have to run another tests.

Also in graphics mode, with a ps/2 keyboard plugged in, when I try to 
bootp (there's a netbsd install server on the network, with dhcpd.conf: 
allow bootp; and filename "netbsd"; in it and tftp serving 
/tftpboot/netbsd which is unzipped 
NetBSD-5.0.2/sgimips/binary/kernel/netbsd-INSTALL32_IP3x.gz), I get this 
error,
No server for :.
[sgi troubleshooting messages...]
[...] bootp [...] could not connect to server

Any help, feedback or experience on those two issues would be appreciated.

Many thanks,
Pierre-Philipp

IP25 Support Question

Richard Yao — 2012-06-22T07:00:54

My department has a SGI Rackmount Power Onyx that was sold to us by SGI
as a Power Challenge. It has IP25 boards. Can it run NetBSD?

I am not subscribed to the list. Please CC me on replies.

Re: 6.0_BETA upgrade fails on Indy / Challenge S

Robert Doerfler — 2012-03-21T19:12:06

Robert Dörfler <rodo< at >zlug.org> schrieb:

btw... i tried to gzip -d and untar pkgsrc.tar.gz

tar: Cannot set file uid/gid of  ()
tar: Cannot set permissions on  (No such file or directory)
tar: Access/modification time set failed on:  (No such file or directory)
tar: Cannot rename  to pkgsrc/audio/gkrellm-volume/Makefile (No such file or directory)
[1]   Segmentation fault (core dumped)gzip: error writing to output: Broken pipe

zucchini# uname -a
NetBSD zucchini. 6.0_BETA NetBSD 6.0_BETA (GENERIC32_IP3x) sgimips

Re: 6.0_BETA upgrade fails on Indy / Challenge S

Robert Dörfler — 2012-03-21T05:30:46

I guess i got the same "unexpected EOF"-Errors on my O2. I installed the
packages via ftp/http and tests.tar.gz failed repeatable. ( even after
choosing a different way to get package ftp/http )